Add MemoizedWebCrawler tutorial #198
Conversation
| * (2) Then it defines a rule for any target that ends with `.tar` extension | ||
| * In the body of the rule, (3) it reads content of a `txt` file with the same prefix and (4) dynamically declares dependency via `need contents` | ||
| * This may trigger building artifacts that match other rules | ||
| * (5) Once the dependencies are met, it spawns `tar -cf` passing all the paths it as CLI arg |
|
|
||
| ## Background | ||
|
|
||
| The core reason why llbuild2fx is powerful is its reliance on content-addressable storage. Think of a tree where each node has a checksum. For each labeled node, the checksum is computed by putting together the checksum of the label, and aggregating the checksum of the children. |
There was a problem hiding this comment.
I think we can make this paragraph better:
In particular, CASTrees (an abstraction very similar to Unix file tree) have a very elegant encoding on top of a content-addressable storage. This fits nicely with the kind of use-cases that a conventional distributed build system is expected to address.
| main = shakeArgs shakeOptions $ do | ||
| want ["result.tar"] -- (1) | ||
| "*.tar" %> \out -> do -- (2) | ||
| contents <- readFileLines $ out -<.> "txt" -- (3) |
There was a problem hiding this comment.
readFile' :: Partial => FilePath -> Action String
Read a file, after calling need. The argument file will be tracked as a dependency.
and
readFileLines :: Partial => FilePath -> Action [String]
A version of readFile' which also splits the result into lines. The argument file will be tracked as a dependency
From https://hackage.haskell.org/package/shake-0.19.8/docs/Development-Shake.html#v:readFile-39-
| public struct FetchHTTP: AsyncFXKey, Encodable { | ||
| public typealias ValueType = FetchHTTPResult | ||
|
|
||
| public static let version: Int = 2 |
There was a problem hiding this comment.
Out of curiosity, what does this field do? (Sorry if this is the wrong place to ask this, but I'm an outsider looking to learn about how llbuild2fx works.) Feel free to email me instead ([email protected]) if you don't want to discuss on the PR.
There was a problem hiding this comment.
what does this field do?
When using llbuild2fx in practice, sometimes you need to update the implementation of computeValue and invalidate values that are currently cached. version field is a mechanism to do that.
llbuild2fx incorporates the version field (plus the sum of version of all keys listed in versionDependencies) in the cache key.
Notice how FetchTitle has the following field:
public static let versionDependencies: [FXVersioning.Type] = [FetchHTTP.self]
When you bump up the version for FetchHTTP, you will also indirectly bump up the version for keys that enlist FetchHTTP in their versionDependencies.
Additionally, if FetchHTTP was not listed in FetchTitle.versionDependencies, then fi.request(FetchHTTP(..)) will throw a runtime error.
Sorry if this is the wrong place to ask this
This is the perfect place to ask these questions. :) I think it'd be useful to add a short description about version in the tutorial. If you needed clarification on the existing content, feel free to comment and I'll incorporate it in the main text.
There is no tutorial out there to explain how to use
llbuild2fx. The current GameOfLife example uses the less popularllbuild2BuildSystemlayer instead ofllbuild2fx.I wrote up a little tutorial in
README.md. I'm intending to fill out the TBD sections prior to merging.