New Chain-Link: Pipe; Simplifying BotAction & Added New #20
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- idea is to provide new way to compose async functions in crawling the web that are dependent on data crawled - idea is to rely on IndexedDB for data in crawling Instagram to reduce dependencies on HTML attributes, CSS classes/ID's etc in harvesting - special BotActionsPipeFactory to create a new kind of chain of resolving async functions except to pass along the previous value
|
What if it were possible to reuse regular Like a higher-order function that wraps each mmm |
|
Idea Rewrite the bot action to return Then handle it in the So the bot actions chain factory must resolve each action with both return cases in mind. Therefore, there is only Maybe the first inject of the |
added 2 commits
April 21, 2020 17:46
- pipe-able, can as a whole return something or not - should the bot actions pipe factory be differentiated further? deprecate?
- need to expand upon the Botmation class actions() method for the simpler yet more powerful BotActionsFactory - to continue above bullet, need to add more testing, while refocusing some deprecatabled tests to cover the BotActionsChainFactory - deprecate the BotActionsPipeFactory ?
|
If Draft 1 is to be committed
|
Codecov Report
@@ Coverage Diff @@
## master #20 +/- ##
===========================================
- Coverage 94.64% 52.25% -42.40%
===========================================
Files 15 25 +10
Lines 168 444 +276
Branches 23 95 +72
===========================================
+ Hits 159 232 +73
- Misses 9 212 +203
Continue to review full report at Codecov.
|
- now if a botaction is used in a pipe, where previous botaction returns a value, but in this case, the dev is not interested in it, they can omit, not think about it, without any breaking changes - previously, the resolved value was injected at the beginning of the injects list, but that can break a botaction method signature, when used without intending on leveraging previous returned value - this way, we can use botactions with or without care for what is previously returned, given the method signature is consistent now
- BotAction now supports optional returning of values to be injected without breaking custom BotAction injects' signature
- waitForNavigation is no longer a higher order method to return a method since it does NOT customize its returned method with any params - Proposal #11
- added defaults to generics of BotAction - expanded upon BotAction with safe defaults as a chain link, then customizable pipable action - added a few actions (to be tested, ie goBack, goFoward, savePDF) - worked on pipe - got IndexedDB pipeable actions working - instagram isGuest now uses IndexedDB data
- having deeper BotAction types like BotFilesAction that specifies injects - getting rid of param piped from BotAction (see BotAction5), for a more specific BotPipeAction that specifies piped value as 1st inject - then a BotAction that relies on options or injects but no piped values can be written without any mention of - downside is the risk of injecting a piped value when used incorrectly without some kind of who is guard or branding of these actions to know when to pipe value or inject injects, etc
- helper method to more easily strongly type your BotActions with less effort
- more work for same level of typing - considering removing 'type?' from BotAction5 for instead using a higher-order Injects BotAction that can be customized ie IndexedDB to set dbName, version, storename?, etc as injects for indexeddb actions, only need strong typing at compile time
- comments left at bottom of bot-actions.interfaces - less is more, stick to some design constraints, and make code safer
- params are all optional, but should not all be ignored with safe fallbacks - idea is you can either inject, pipe, or use the factory method to customize what is set in IndexedDB
- building, all new code being tested in examples/instagram - indexedDBStore higher order botaction WIP - setIKeyVal3, getIKeyVal3 methods (upgraded versions of setIndexDBStoreDataKeyValue, etc) are WIP - idea for tubing in layers injects, so higher order chains/pipes injects are injected at the end, with most recent injected 1st - so in theory, in the future, could expand upon higher order functions to support nesting - ie IndexedDB(dbName, dbVersion)((IndexedDBStore(Store)(...actions))
- the get/set methods with BotAction5 syntax is working - test 5 is failing, tbd - BotAction5 > BotAction10 for now, less code in argument typing for majority cases - need to add a files()() for injecting BotOptions -> BotFileOptions
- expanded for new Piping logging behavior and spacing vertically ie log() adds a line break at the end, even when given nothing to log
- refactored logPipeValue() for easier readability and improved coverage
- integration test for wait() on sleep()
- missed edge case now tested in getFileUrl()
- integration test for getting key/value from IndexedDB
- adjusted unpipeInjects() to return pipeValue at the start of the array
- added the ability to set an expected number of injects in
unpipeInjects to fill whats missing with what w/e, default undefined
- can now strongly type the pipe value through a generic and its safer
for situations where the number of injects is ultimately unknown
- makes it so devs using it dont have to test their injects as much
for more complex situations
- set method integration test - updated the mocking of console.log and error to restore original functionality afterAll suite tests run
- higher order indexed-db action testing, ended up finding a bug where parent injects were not injected, maybe but in pipe()()() tbd - e2e test is failing for indexed-db on an error, reported on github jest-puppeteer, linked in comments
- recently added type gaurds unit testing
- unit-testing chainRunner, pipeRunner, and pipeActionOrActions
- unit testing chain()
- pipe() unit-testing covers all cases
- assemblyLine() unit-testing - missed coverage testing in chain()
- added additional constraint for chain() that the awaited result is no other than undefined
- chainRunner() extended testing to add constraint that a resolved chainRunner is no other value than undefined
- removed tests for bot-actions-chain.factory - deleted bot-actions-chain and /factories directory, no longer needed - such functionality is now available as an assembly-line BotAction
- indexedDBStore() runs an assembly-line so needs to await inject()()() - remove dead url
|
As of a few commits ago, the minimum threshold for testing coverage in this Project has been met As of now, only 1 helper file has lines of code not covered Close to 100%: |
|
Before merge:
Next PR:
Future:
|
- verifying function running of all BotAction's in chainRunner()
- missed getPage() on class.spec - expanding upon testing of indexed-db helpers, refining their functionality
- missed edge case in asyncConstructor for class.spec - unit testing getIndexedDBStoreValue(), seems I have more to understand with IndexedDB's nature as the onerror handlers didn't work as expected
- added the ability to not specify the database version number since not providing it results in a safe route of latest version, helpful - updated relevant testing - the injects params and higher order params order for IndexedDB BotAction's have changed in response to this
- had to ignore a line of code, but left an assertion that covers it, but for whatever reason, the coverage report misses it - the code is the ability to create a new store during a db version upgrade (version number bigger than before), tested with comments
|
Test coverage is now 💯🥳 |
This was referenced Jul 29, 2020
Closed
- separate test file to test an inject()() theory about nesting since this test requires assemblyLine() not mocked, it was separated - can put other theories of nesting, etc in here - might move in future, reconsolidate
- reviewed todo's in comments - some minor cleaning
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Concept work
BotActionsPipeFactorycreates a chain-link, which is a function that returnsPromise<void>composesBotPipeActionfunctions in a pipe, similar to the main bot actions chainpipes pass the resolved data of the previous function in the params of the next functionBotPipeActionFactoryreturns aBotPipeAction<T>generic designates the type of the value being passed alongBotPipeActionis anasyncfunction that returnsPromise<T>Draft
Generic
BotActionsFactoryif pipe'ing then use the firstinjectsparam for the piped dataBasic IndexedDB BotAction's to pipe data
how do we want to
normalize,map, etc these pipe's?update remaining BotAction's for new interface
BotAction5, then switch namestesting
todomessages regarding testing