tech-lessons.in
learnings by a developer
- -Share on Facebook
- -Tweet
-
+article that explores integration testing using Localstack for our serverless application.
diff --git a/en/blog/bitcask/index.html b/en/blog/bitcask/index.html
index 68e82e5..5aa9b73 100644
--- a/en/blog/bitcask/index.html
+++ b/en/blog/bitcask/index.html
@@ -221,9 +221,9 @@
Hint files are smaller than data files because they don’t contain values. Hence, recovery is done from hint files instead of data files.
Conclusion
Bitcask has a relatively simple data structure compared to LSM , and it offers some great positives:
- High write throughput: it performs sequential disk writes, offering high write throughput.
- Low latency per item read or written: all the operations require a single disk seek.
- Ease of backup and restore: since the data files are immutable, backup can be taken by copying the directory. Restoration requires nothing more than placing the data files in the desired directory.
- Crash friendliness: As the data files and the commit log are the same things in Bitcask, recovery is trivial.
- A relatively simple model to understand.
Bitcask model has a set of challenges:
- High memory usage: Bitcask stores all the keys in RAM. Thereby the memory usage is high. However, if the keys can be partitioned the problem can be reduced to some extent.
- Huge number of open OS file handles: Get operation in Bitcask can happen from both the active and inactive segment files, and there could be large number of inactive segment files in a single Bitcask instance. To perform a read from an inactive segment, we might want to keep the file pointer of the inactive segment file open. This can result in too many open OS file handles.
Code
The code for this article is available here -.
References
- -Share on Facebook
- -Tweet
-
+.
References
diff --git a/en/blog/bloom_filter/index.html b/en/blog/bloom_filter/index.html index 695cbb5..39d4d5c 100644 --- a/en/blog/bloom_filter/index.html +++ b/en/blog/bloom_filter/index.html @@ -191,9 +191,9 @@ .Mention
Thank you, Debasish Ghosh for reviewing the article and providing feedback.
References
diff --git a/en/blog/code_without_tests/index.html b/en/blog/code_without_tests/index.html index 20fe5aa..f3ab275 100644 --- a/en/blog/code_without_tests/index.html +++ b/en/blog/code_without_tests/index.html @@ -104,9 +104,9 @@ get better at things before it all comes crashing down.Probabilistic data structures provide approximate answers to queries about a large dataset rather than exact answers. These data structures are designed to handle large amounts of data in real-time by making trade-offs between accuracy and time and space efficiency. ↩︎
LSM Tree -A log-structured merge tree (LSM tree) is a data structure typically used when dealing with write-heavy workloads. The write path is optimized by performing sequential writes. ↩︎
If a team believes there is delivery pressure today and tests can be added tomorrow, then the team needs to be sure of one thing - “That tomorrow is never coming”.
Mentions
I would like to thank Gurpreet Luthra , Unmesh Joshi and Sunit Parekh -for providing feedback on the article. Thank you Gurpreet, Unmesh and Sunit.
These commands make a few assumptions -
- AWS credentials are already configured on the host machine
- AWS user has the right to create various AWS resources including IAM roles
dist/directory which will be deployed on an S3 bucket (bootstrap creates for us) when we executecdk bootstrap, already exists
It will take sometime for stack to be created which will consist of
lambda function, DynamoDB table, API gateway and all the necessary IAM roles.Once our stack is created, make an entry in the
orderstable, hit the public API endpoint which will look likehttps://rest-api-id.execute-api.ap-south-1.amazonaws.com/dev/orders/OrderIdand enjoy the output.That’s it, our stack is deployed and our application is up and running 😁
Conclusion
Relationship between CDK and CloudFormation can be summarised as -
In this article we were able to code our infra using CDK, write tests for our infra and deploy the same. Let’s take a look at some advantages of using CDK -:
- Resources can be modeled in Object-Oriented manner
- High level abstractions can be defined and published within the team or company
- Infrastructure can be built as library
- Infrastructure code can be tested
- IDE’s code completion can be leveraged
- Programming language constructs like if statements, for-loops, etc. can be used when defining infrastructure
We have finally come to end of our Serverless Journey series. Hope you enjoyed it.
The result for cache-hits in CacheD is available here .
That’s it. We have all the building blocks needed to build an in-memory LFU cache.
Code
The source code of CacheD is available here and the crate is available here -.
Relevant research papers
References
- -Share on Facebook
- -Tweet
-
+.
Relevant research papers
References
diff --git a/en/blog/diving_into_java_bytecode/index.html b/en/blog/diving_into_java_bytecode/index.html index b0189d2..f98027a 100644 --- a/en/blog/diving_into_java_bytecode/index.html +++ b/en/blog/diving_into_java_bytecode/index.html @@ -226,9 +226,9 @@ value ofnumberandthisreference on the stack. We needthisreference to be able to get the value of instance variablennumber <= this.n iload_2 and aload_0 should be able to copy the value of numbervariable from slot 2 andthisreference from slot 0 on the stackWe should be able get the value of class field nthis.n getfield should be able to help here. It pops the class instance to get the field value. The field value goes on the stack. Now, our stack contains value of number and n Perform the required comparison. If the condition indicates exit from the loop, return the value present on the stack top number <= this.n if_icmpgt does the integer comparison. It pops the top 2 integer values from the stack and does the comparison (number > n). If condition returns true, it takes an argument which is the instruction offset to jump to If the condition indicates loop continuation, load the value of sumandnumbervariable to be able to perform additionsum + number iload_1 and iload_2 should do it. Now our stack has 2 values which are ready for addition Perform addition sum + number iadd does the integer addition. Pops the top 2 values from the stack and puts the result back in the stack Assign the result of addition to the variable sumsum = sum + number istore_1 would do the job. It takes the value from stack top and assign the value in variable sum Increment the value of numbernumber = number + 1 iinc does integer increment and takes 2 parameters. First one is the LocalVariableTable slot and other one is the increment. It is one of the opcodes that does not work with stack. It increments the value at a specific slot in LocalVariableTable Repeat steps NA goto is the opcode which transfers the control to a specific instruction set Following diagram represents the overall execution of
summethod -
Summary
Let’s conclude with some key takeaways -
- javap provides a human-readable format of class file
- Each opcode in a bytecode is represented by 1 byte
- Each opcode is prefixed with a letter indicating the data type the opcode will work with
- Most of the opcodes work with the stack which means before they operate, values need to be brought on the stack
- Opcode like iinc works with LocalVariableTable instead of working with values on the stack
- Opcodes like invokevirtual, invokespecial refer to an entry in the constant pool to resolve the method that needs to be invoked
- Some opcodes also have shortcuts. eg; iconst_0 pushes 0 on the stack without taking any argument. It could have been designed to take an argument but that would have meant the total instruction size will be greater than 1 byte (1 byte for the opcode and another byte for the argument). In order to avoid -increasing the size of the instruction, it is designed in a shortcut form
Hope it was meaningful. Appreciate the feedback.
References
- Advanced Java Bytecode Tutorial
- Java Bytecode: Using Objects and Calling Methods
- Java Bytecode Crash Course
- Optimizing Java - practical techniques for improving JVM application performance
Hope it was meaningful. Appreciate the feedback.
References
- Advanced Java Bytecode Tutorial
- Java Bytecode: Using Objects and Calling Methods
- Java Bytecode Crash Course
- Optimizing Java - practical techniques for improving JVM application performance
Wrap Up
We believe the MVP is done and features like flipping at runtime and supporting database-driven feature flips are in the pipeline. For any custom flip condition, one could go ahead and use
@FlipOnSpringExpressionwith your custom spring bean to determine the flip condition. If you want to have a look at the code or even want to contribute, you can check out Flips -.Feel free to share any feedback.
- -Share on Facebook
- -Tweet
-
+.
Feel free to share any feedback.
diff --git a/en/blog/invest_in_blogging/index.html b/en/blog/invest_in_blogging/index.html index 8abc63c..52c08ac 100644 --- a/en/blog/invest_in_blogging/index.html +++ b/en/blog/invest_in_blogging/index.html @@ -7,9 +7,9 @@ 927 wordsTable of contents
We often have a lot to share with people, this could be our learnings, our opinions and our experiences. There are times when we feel the need to get our ideas validated or get feedback from people. These are definitely some of the reasons to invest in blogging and connect with community.
Let’s see some reasons for investing in writing blogs.
Help people learn from your learning journey
There are times when we often feel “it would have been great if someone had written an article to explain a concept”, start writing if you have had this feeling.
Learning is like climbing a rock. While climbing, we always look at the tip of the rock just to realize it is too far away. What is also important is to realize that there are people who might have just started this journey and your “learning journey” could go a long way in helping them.
Help people learn from your mistakes
Share your mistakes with the community. We as a community learn from each other’s mistakes and experiences and these things are really valuable.
Your blog on “Failing with Microservices” could help me in avoiding some mistakes or at least rethink my design if I am starting with microservices.
Get feedback from community
There are a lot of things which help us grow as an individual and one of them is feedback or I should say “Constructive Feedback”. Write to get feedback from community, to get their thoughts, to hear their experiences and to learn from all of these. Let’s see how this could work.
Say, I am very excited to use Coroutines to build reactive streams in my next project and I share an article “Being Reactive with Kotlin Coroutines” which talks about the basics of Coroutines and abstractions like “Channel” to implement reactive streams.
This article receives a lot of feedback and one of the feedback says -
Hey, nicely put. I would suggest you to check this link. It says - “There is no way to receive the elements from a channel again. The channel is closed when the producer coroutine is over and the attempt to receive from it again cannot receive anything." -You might also want to take a look at kotlinx-coroutines-rx2.
Now, this is important as it helps me understand a lot of dimensions including backpressure, hot and cold observables which I had not considered. Thanks to the feedback, I got pointed in the right direction.
. . .
Investment is tricky and one expects a return from every investment. Let’s see the overall “return over investment” in blogging.
Solidifies your understanding
Like we learn when we teach people, we also learn when we share our ideas with people. Blogging helps in solidifying our understanding and the reason I say this is -
We try to communicate our ideas in the simplest possible manner to our readers. In order to do this, we choose to take small steps and each of these steps is well thought of and analyzed. Each step in turn teaches us something which improves our understanding.
We were talking about DSLs in Kotlin in one of the workshops, and I happened to like the way that topic was built - from lambdas to extension function to lambdas with receiver to invoke function.
I decided to share the same in an article Kotlin DSLs: The Basics and if I look back, I realize these two things - a workshop and an article have really helped me understand Kotlin DSLs well.
Increases your confidence
You are not afraid of reaching out to people, and you are not the same person you used to be who would think “should I share this, people would already know it”, “this tech was released 5 years back and I am writing about it now, does it make sense?”. You become someone who would share his/her ideas with confidence.
Challenges you to write better every time
You challenge yourself to write better every time. You tend to experiment with different styles of writing in an attempt to communicate your ideas clearly and connect with people better.
Makes you a better articulator of thoughts
You tend to wear a writer’s hat every time you sit to share something. An attempt is made is to talk to the readers through your article which acts like a story. You read your article hundreds of times in an attempt to articulate better. All this does is make you a better articulator of thoughts.
Builds your network
Investment in blogging is a great way to build network, you get to know people and people get to know you.
Networking is very powerful and truly magical, it can surprise you with a lot of opportunities which you might not imagine. You might get to speak at conferences, work with people that you follow and many more.
Investment in blogging is really a simple way to build network !!
Builds your brand
Investment in blogging acts as a great tool to build your and your organization’s brand.
“Return over investment in blogging” looks promising, but we need to be aware of the fact that a return might not be immediate for an investment.
Making an investment is the first step and usually the most difficult step, rest is all about return :)
Take your first step with blogging, share your ideas / opinions / thoughts with the community. It is a great tool which does a lot of magic, has got great return and more importantly “it is fun”.
Invest in blogging.Summary
- -Share on Facebook
- -Tweet
-
+You might also want to take a look at kotlinx-coroutines-rx2.
Now, this is important as it helps me understand a lot of dimensions including backpressure, hot and cold observables which I had not considered. Thanks to the feedback, I got pointed in the right direction.
. . .
Investment is tricky and one expects a return from every investment. Let’s see the overall “return over investment” in blogging.
Solidifies your understanding
Like we learn when we teach people, we also learn when we share our ideas with people. Blogging helps in solidifying our understanding and the reason I say this is -
We try to communicate our ideas in the simplest possible manner to our readers. In order to do this, we choose to take small steps and each of these steps is well thought of and analyzed. Each step in turn teaches us something which improves our understanding.
We were talking about DSLs in Kotlin in one of the workshops, and I happened to like the way that topic was built - from lambdas to extension function to lambdas with receiver to invoke function.
I decided to share the same in an article Kotlin DSLs: The Basics and if I look back, I realize these two things - a workshop and an article have really helped me understand Kotlin DSLs well.
Increases your confidence
You are not afraid of reaching out to people, and you are not the same person you used to be who would think “should I share this, people would already know it”, “this tech was released 5 years back and I am writing about it now, does it make sense?”. You become someone who would share his/her ideas with confidence.
Challenges you to write better every time
You challenge yourself to write better every time. You tend to experiment with different styles of writing in an attempt to communicate your ideas clearly and connect with people better.
Makes you a better articulator of thoughts
You tend to wear a writer’s hat every time you sit to share something. An attempt is made is to talk to the readers through your article which acts like a story. You read your article hundreds of times in an attempt to articulate better. All this does is make you a better articulator of thoughts.
Builds your network
Investment in blogging is a great way to build network, you get to know people and people get to know you.
Networking is very powerful and truly magical, it can surprise you with a lot of opportunities which you might not imagine. You might get to speak at conferences, work with people that you follow and many more.
Investment in blogging is really a simple way to build network !!
Builds your brand
Investment in blogging acts as a great tool to build your and your organization’s brand.
“Return over investment in blogging” looks promising, but we need to be aware of the fact that a return might not be immediate for an investment.
Making an investment is the first step and usually the most difficult step, rest is all about return :)
Take your first step with blogging, share your ideas / opinions / thoughts with the community. It is a great tool which does a lot of magic, has got great return and more importantly “it is fun”.
Invest in blogging.Summary
diff --git a/en/blog/invoking_c_from_go/index.html b/en/blog/invoking_c_from_go/index.html
index 2d024f8..064dcf7 100644
--- a/en/blog/invoking_c_from_go/index.html
+++ b/en/blog/invoking_c_from_go/index.html
@@ -178,9 +178,9 @@
t.Fatalf("Expected %v, received %v", expected, values)
}
}
-
That is it, run the test and get all the values from linked list.
Code
The code for this article is available here
References
- Calling C code from go
- Golang slices and C pointers
- What does 1-30c-yourtype do exactly in cgo
- ld: symbol(s) not found for architecture x86_64
That is it, run the test and get all the values from linked list.
Code
The code for this article is available here
References
- Calling C code from go
- Golang slices and C pointers
- What does 1-30c-yourtype do exactly in cgo
- ld: symbol(s) not found for architecture x86_64
The operator function
invoke()can come in handy while building DSL.Conclusion
- Kotlin provides a first class support for DSL which is type safe
- One can create a DSL in Kotlin using -
- Lambda as function parameters
- Lambda with receiver as function parameter
- Operator function
invokealong with lambda with receiver as function parameter
References
- Kotlin In Action
The operator function
invoke()can come in handy while building DSL.Conclusion
- Kotlin provides a first class support for DSL which is type safe
- One can create a DSL in Kotlin using -
- Lambda as function parameters
- Lambda with receiver as function parameter
- Operator function
invokealong with lambda with receiver as function parameter
References
- Kotlin In Action
Immutable Classes: It would be good to see something like a
readonlyorimmutablemodifier to create an immutable class. The below-mentioned code snippet is simply a thought (not available in Kotlin or Java).//Hypothetical immutable class User(private val name: String, private val id: Int) -Conclusion
As developers, we will always make mistakes (skipping NULL checks, mutating a collection, etc.), but providing features at the language level that can stop these mistakes will make our lives easier and prevent mistakes.
Conclusion
As developers, we will always make mistakes (skipping NULL checks, mutating a collection, etc.), but providing features at the language level that can stop these mistakes will make our lives easier and prevent mistakes.
diff --git a/en/blog/lets_deal_with_legacy_code/index.html b/en/blog/lets_deal_with_legacy_code/index.html index a78fa21..9f3eb7c 100644 --- a/en/blog/lets_deal_with_legacy_code/index.html +++ b/en/blog/lets_deal_with_legacy_code/index.html @@ -243,9 +243,9 @@ } return totalAmount; } -
I cheated again. Did lot more than what I should have done, renamed methods to be text*, duplicated for loops (over rentals) to calculate the total amount, repeated the same in the
textBody()method.Is that justified? Well, how many rentals do we expect to have for a customer? What is the cost of iterating over them twice? If it is not significant, go ahead and use it. What does it give me? Look at the
statement()(renamed astextStatement()) method now.Jessica> Now, we are done with refactoring. We can introduce the HTML statement functionality now.
Conclusion
Jessica and Scott went on to implement the HTML functionality (with tests) and they did a lot to clean up the existing code. The code is much more understandable that it used to be.
They might not have cleaned up everything, but they have left a great deal of understanding trace for others to follow.
They followed Cover and Modify, Boy Scout rule, Refactoring cycle and refactored enough to finish the new functionality, in-short dealt with Legacy code professionally.
References
- Refactoring Improving The Design Of Existing code
- Refactoring Catalog
I cheated again. Did lot more than what I should have done, renamed methods to be text*, duplicated for loops (over rentals) to calculate the total amount, repeated the same in the
textBody()method.Is that justified? Well, how many rentals do we expect to have for a customer? What is the cost of iterating over them twice? If it is not significant, go ahead and use it. What does it give me? Look at the
statement()(renamed astextStatement()) method now.Jessica> Now, we are done with refactoring. We can introduce the HTML statement functionality now.
Conclusion
Jessica and Scott went on to implement the HTML functionality (with tests) and they did a lot to clean up the existing code. The code is much more understandable that it used to be.
They might not have cleaned up everything, but they have left a great deal of understanding trace for others to follow.
They followed Cover and Modify, Boy Scout rule, Refactoring cycle and refactored enough to finish the new functionality, in-short dealt with Legacy code professionally.
References
- Refactoring Improving The Design Of Existing code
- Refactoring Catalog
Hold on, did we just copy the output generated by the code and placed into our test. Yes, that is exactly what we did.
We aren’t trying to find bugs right now. We are trying to put in a mechanism to find bugs later, bugs that show up as differences from the system’s current behavior. When we adopt this perspective, our view of tests is different: They don’t have any moral authority; they just sit there documenting what the system really does. At this stage, it’s very important to have the knowledge of what the system actually does.
Question> What is the total number of tests that we write to characterize a system?
Answer> It’s infinite. We could dedicate a good portion of our lives for writing case after case for any class in a legacy code.
Question> When do we stop then? Is there any way of knowing which cases are more important than others?
Answer> Look at the code we are characterizing. The code itself can give us ideas about what it does, and if we have questions, tests are an ideal way of asking them. At that point, write a test or tests that cover good enough portion of the code.
Question> Does that cover everything in the code?
Answer> It might not. But then we do the next step. We think about the changes that we want to make in the code and try to figure out whether the tests that we have will sense the problems that can happen. If they won’t, we add more tests until we feel confident that they will.
How Much To Refactor
There is so much to refactor in legacy code, and we can not refactor everything. In order to answer this we need to go back to understanding our purpose of refactoring the legacy code. We want to refactor legacy code to leave it cleaner than what it was when it came to us and to make it understandable for others. With that said, we want to make the system better keeping the focus on the task. We don’t want to go crazy with refactoring trying to improve the whole system in a few days. What we want to do is refactor the code that comes in our way of implementing any new change. We will try and understand this better with an example in the next article.
Let’s move on to the next article -that explains how to deal with legacy code.
References
- Working Effectively with Legacy code
- -Share on Facebook
- -Tweet
-
+that explains how to deal with legacy code.
References
- Working Effectively with Legacy code
LocalStack exposes an environment variable
LOCALSTACK_HOSTNAMEwhich is available inside the docker process that refers to the host machine.That’s it run all the tests
npm tand see them pass 😁Summary
We used LocalStack to test our application. Everything is available here .
Here is a quick glimpse of the sequence of events that happen when the integration tests are executed.
Let’s move on to our last article -and see everything in action on an AWS account.
MergerIterator is available here .
Conclusion
We have finally reached here :).
LSM-tree based storage engines typically include the following data structures:
- On-disk log file (WAL) to persist the writes.
- In-memory memtable(s).
- On-disk files organized in levels.
LSM-trees offer higher write throughput because the writes are always sequential in nature, but reads are not so great because LSM-trees may have to scan multiple files or portions of multiple files. One idea to improve the reads in LSM-trees is to reduce the size of SSTable files and cache some layers of SSTables.
When designing a storage engine for SSDs, we should consider SSD characteristics including:
- SSDs can wear out through repeated writes, the high write amplification in LSM-trees can significantly reduce the device lifetime.
- SSDs offer a large degree of internal parallelism
The core ideas of WiscKey include: separating values from keys in the LSM-tree to reduce write amplification and leveraging the parallel IO characteristic of SSD.
I hope the article was worth your time. Feel free to share the feedback.
Mention
Thank you, Unmesh Joshi -for reviewing the article and providing feedback.
References
diff --git a/en/page/about/index.html b/en/page/about/index.html index 121b52d..cccb938 100644 --- a/en/page/about/index.html +++ b/en/page/about/index.html @@ -21,10 +21,7 @@ as the metadata store - badger as the key/value storage engine
- raft (/multi-raft) as the consensus algorithm and
- two-phase commit -with serial isolation level
Let’s connect
Feature sendEmail is enabled if the property
feature.send.emailis set totrue.I have a separate blog -about this library.
