|
| 1 | +--- |
| 2 | +skip: True |
| 3 | +prompt: Write instructions on how to do Workflow Scouting with Ailly. |
| 4 | +--- |
| 5 | + |
| 6 | +# How to Scout with Ailly |
| 7 | + |
| 8 | +## Prereq: Install and run Ailly |
| 9 | + |
| 10 | +1. On bash, easiest way is probably an alias: `alias ailly=npx @ailly/cli` |
| 11 | + - which will probably need `npm install -g @ailly/cli` |
| 12 | + - and can then be updated with `npm install -g @ailly/cli@{version}` |
| 13 | +1. Choose your engine, probably bedrock. |
| 14 | + - Export an environment variable: `export AILLY_ENGINE=bedrock`. |
| 15 | + - Ensure you're using your AWS account (`ada` or copying the access keys from isengard) |
| 16 | + |
| 17 | +## Readme |
| 18 | + |
| 19 | +The first step is getting a feel for Ailly and working with it. We'll have it make a README for the workflow, edit the README ourselves for fine detailing, and use it in "assist" or "tmp" or "macro" mode (I need a good name for this) to do bulk editing tasks. |
| 20 | + |
| 21 | +1. WITHOUT using an LLM, do the Workflow process. Meet a SME, develop a high level plan for what the workflow should do. |
| 22 | +2. Create a new workspace for your scout in the `workflow` folder. |
| 23 | +3. Create a new folder for doing Ailly work, `workflow/scout_name/content`, and cd into it. |
| 24 | +4. Add a `.aillyrc` file with: |
| 25 | + - A greymatter head: |
| 26 | + ``` |
| 27 | + --- |
| 28 | + combined: true |
| 29 | + --- |
| 30 | + ``` |
| 31 | + - A level-setting prompt (Coming soon!) to guide output format. |
| 32 | + - The summary of the Code Examples team. |
| 33 | + - A summary of the service you're writing the workflow for. |
| 34 | + - The summary of the workflow. |
| 35 | + - A list of the API calls you expect to make. |
| 36 | +5. Add a file, `01_README.md`, with a greymatter head: |
| 37 | + ``` |
| 38 | + --- |
| 39 | + prompt: Create a README.md for this project |
| 40 | + --- |
| 41 | + ``` |
| 42 | + - at any (and every?) point in this process, experiment with your own prompting! |
| 43 | +6. Run ailly: `ailly 10_README.md` |
| 44 | +7. Review the output. Correct it as desired. |
| 45 | +
|
| 46 | + - If there's a thing you want to change, you can use Ailly like a macro |
| 47 | + - Let's say Ailly creates a list of the API calls to use, but doesn't include links |
| 48 | + - Create a folder, `tmp` |
| 49 | + - Create an aillyrc, `tmp/.aillyrc`, with |
| 50 | + ``` |
| 51 | + --- |
| 52 | + isolated: true |
| 53 | + --- |
| 54 | + ``` |
| 55 | + - Create a file, `tmp/links.md` |
| 56 | + - Add this head: |
| 57 | +
|
| 58 | + ``` |
| 59 | + --- |
| 60 | + prompt: | |
| 61 | + Reformat this list into links to the documentation. For instance, the SendEmail item should link to https://docs.aws.amazon.com/ses/latest/APIReference-V2/API_SendEmail.html. |
| 62 | +
|
| 63 | + SendEmail |
| 64 | + CreateEmailIdentity |
| 65 | + [paste the rest of your items] |
| 66 | + --- |
| 67 | + ``` |
| 68 | +
|
| 69 | + - Run Ailly: `ailly tmp/links.md` |
| 70 | + - Review the output, which should now have |
| 71 | + ``` |
| 72 | + * [SendEmail](https://docs.aws.amazon.com/ses/latest/APIReference-V2/API_SendEmail.html) |
| 73 | + * [CreateEmailIdentity](https://docs.aws.amazon.com/ses/latest/APIReference-V2/API_CreateEmailIdentity.html) |
| 74 | + * [The rest of your links] |
| 75 | + ``` |
| 76 | + - If this doesn't work the first time, rerun and see if it helps. |
| 77 | + - If it doesn't a second time, play with the prompt until it does what you want. |
| 78 | + - Paste the output back into the `01_README.md` document. |
| 79 | +
|
| 80 | +## First Spec |
| 81 | +
|
| 82 | +At this point, you should have a good start on the README, which we can use to have Ailly write the spec. From current experiments, LLM generation works best with ~500 word output "chunks", so we start planning our Ailly calls around that. It also works best having additional context, so we'll get that first. |
| 83 | +
|
| 84 | +1. Help Ailly get API docs for the service. |
| 85 | +
|
| 86 | + - Create a new file, `tmp/get_api.md` |
| 87 | + - Add a prompt: |
| 88 | +
|
| 89 | + ``` |
| 90 | + --- |
| 91 | + prompt: | |
| 92 | + Write a (bash or powershell or batch (or python?)) script to download the API docs for each API used in this workflow. |
| 93 | +
|
| 94 | + An API doc link looks like https://docs.aws.amazon.com/ses/latest/APIReference-V2/API_SendEmail.html |
| 95 | +
|
| 96 | + After downloading, pass the HTML files first through `pup` to select '#main-content', then use `html2text -nobs -utf8` to get just the main text of the document. |
| 97 | +
|
| 98 | + Put the downloaded text in a file, 10_{ApiName}.md, with a greymatter header that has a property `skip: true`. |
| 99 | +
|
| 100 | + For example, `echo "---\nskip: true\n---" > 10_SendEmail.md\ncurl https://docs.aws.amazon.com/ses/latest/APIReference-V2/API_SendEmail.html | pup "#main-content" | html2text -nobs -utf8 >> 10_SendEmail.md` |
| 101 | + --- |
| 102 | + ``` |
| 103 | +
|
| 104 | + - This kind of prompt can take a bit of work to get right. In my experience, I spend a couple minutes digging into structure of the document, then use the command line tools like `pup`, `jq`, `yq`, `html2text`, `pandoc` |
| 105 | + - It might be faster to just copy/paste it a handful of times, but that's not as fun |
| 106 | + - Why spend 2 minutes doing repetitive tasks when we could spend 15 minutes learning how to make the LLM do the repetitive task for us? |
| 107 | + - To run the commands it generates, I just copy/paste them into my terminal. #yolo |
| 108 | + - Real #yolo will come maybe next quarter when I add a tools module to ailly to give it direct shell access |
| 109 | + - However you do it, make sure you end up with a few `10_API.md` files in your `content/` folder. These will be used by the remaining prompts. |
| 110 | +
|
| 111 | +2. Create the first part of the spec, `30_SPECIFICATION.md` (I left the 20s available for future use...) |
| 112 | +
|
| 113 | + - Contents: |
| 114 | +
|
| 115 | + ``` |
| 116 | + --- |
| 117 | + prompt: | |
| 118 | + Write a specification for this workflow. |
| 119 | +
|
| 120 | + [Lots of instructions for how you want it to handle the workflow!] |
| 121 | +
|
| 122 | + [How should it handle input? Prompt, or variables? Team says prompt...] |
| 123 | +
|
| 124 | + [Share what works and what doesn't! We can build a library of these!] |
| 125 | +
|
| 126 | + [Example:] |
| 127 | +
|
| 128 | + The specification is independent of any programming language, and should enable any programmer competent with programming using any published AWS SDK to follow along. It must specify the API calls to make, and it must include the parameters to send. It should describe the parameters in a list format. Implementations will use the specific SDKs, so it does not need to specify URL calls to make, only the API calls and the request parameters to include. It must specify the exact environment variable names and files to use when referring to runtime data. |
| 129 | + --- |
| 130 | + ``` |
| 131 | +
|
| 132 | + - Run Ailly! `ailly 30_specification.md` |
| 133 | + - As we keep specifying the file name, Ailly will load the entire directory before this file as context, but only generate this file as output. |
| 134 | + - This output will, keeping with the "500 words", probably be pretty close to the README but a bit more formal. That's OK, we're going to build from here. |
| 135 | + - Iterate the prompt until you get a result you like. Keep copies or git commits, as you prefer, if you want. |
| 136 | + - When you have one you like, edit it a bit. |
| 137 | +
|
| 138 | +3. Create specific parts of the spec. If the workflow has five "parts", create `31_PART_1.md`, `32_PART_2.md`, etc. (Recommend replacing `PART_1` with the short name of the part or step.) |
| 139 | +
|
| 140 | + - For each, this is the prompt I started with: |
| 141 | + ``` |
| 142 | + --- |
| 143 | + prompt: | |
| 144 | + Describe the exact SESv2 API calls and parameters for this step. |
| 145 | + ## Prepare the Application |
| 146 | + --- |
| 147 | + ``` |
| 148 | + - This prompt kinda sucked, TBH, but it got the job done. Lots of room for improvement here. |
| 149 | + - Run Ailly: `ailly 31_PART_1.md` (or `ailly 3{1,2,3}*.md`). |
| 150 | + - Iterate! |
| 151 | +
|
| 152 | +4. If the spec needs sample files, have Ailly & Claude make them! |
| 153 | +
|
| 154 | + - `50_SAMPLE_FILES.md` |
| 155 | + ``` |
| 156 | + --- |
| 157 | + prompt: | |
| 158 | + List and describe the sample files that this workflow will need at runtime. |
| 159 | + --- |
| 160 | + ``` |
| 161 | + - `51_SAMPLE_FILE_A.md` |
| 162 | + ``` |
| 163 | + --- |
| 164 | + prompt: | |
| 165 | + Create [Sample File A] |
| 166 | + --- |
| 167 | + ``` |
| 168 | + - etc |
| 169 | +
|
| 170 | +5. Consolidate these files |
| 171 | + - Maybe in the future Ailly can have file system access and know how to issue instructions to combine the various in-progess files, but for now I just open the handful of files and copy/paste from `content/01_README.md` to `README.md`. |
| 172 | + - Open a PR and review the workflow spec |
| 173 | +
|
| 174 | +# First Language |
| 175 | +
|
| 176 | +## Structure |
| 177 | +
|
| 178 | +1. Create a folder for the language, say, `python`, and add an `.aillyrc`: |
| 179 | +
|
| 180 | + ``` |
| 181 | + You are a Python programmer, using Python 3.9. |
| 182 | + |
| 183 | + <examples> |
| 184 | + <example type="main with argument handling"> |
| 185 | + ...python... |
| 186 | + </example> |
| 187 | + |
| 188 | + <example type="input request"> |
| 189 | + ...python... |
| 190 | + </example> |
| 191 | + |
| 192 | + <example type="error handling"> |
| 193 | + ...python... |
| 194 | + </example> |
| 195 | + |
| 196 | + <example type="sdk call"> |
| 197 | + ...python... |
| 198 | + </example> |
| 199 | + |
| 200 | + <example type="pagination"> |
| 201 | + ...python... |
| 202 | + </example> |
| 203 | + </examples> |
| 204 | + |
| 205 | + |
| 206 | + Imports should be sorted. [Additional notes as necessary] |
| 207 | + ``` |
| 208 | +
|
| 209 | + You may or may not include snippets for these, but they should help. If there are other examples you find help a lot, let the team know! |
| 210 | +
|
| 211 | + Anthropic claims [claude does best with XML tags](https://docs.anthropic.com/claude/docs/long-context-window-tips)? But I've seen it do fine with markdown? So I'm playing with both, preferring Markdown for quick self contained things, and XML for longer/larger/more detailed examples. |
| 212 | +
|
| 213 | + - When running ailly, run it from the folder with the original `.aillyrc` - `ailly python/20_PLAN.md`. |
| 214 | +
|
| 215 | +1. Copy the current files. Ailly uses `.aillyrc` files going up, but only includes files in the current folder for the current context. |
| 216 | + - Copy the README.md and SPECIFICATION.md consolidated files to `01_README.md` and `02_SPECIFICATION.md`. |
| 217 | +1. Find the language-specific API documentation, and put it into `10_[API_CALL].md`. |
| 218 | + - Add greymatter to each with `skip: true` (don't want to be regenerating these.) |
| 219 | +1. Create a `20_PLAN.md`. Your plan should have a prompt asking for the implementation files and class & method stubs, but not full implementations |
| 220 | + - Generate the plan. This might take a few passes. Maybe add some examples for how you want it structured. Eventually you should come out with an outline for the project you're satisfied with. |
| 221 | + - You can ask it to make the plan in the form of a shell script that would create the files it wants - this can be helpful if you know you'll need a few files for different parts. |
| 222 | + - Maybe create `21b_PLAN_SUPPLEMENTAL.md` with a `skip: true` and prompt: "When implementing methods, only implement the method you're currently instructed to implement." There may be other supplemental details as well. |
| 223 | + - The important part here is to remind you you can slip in additional details for all downstream instructions in a few different ways. |
| 224 | + - When running ailly, run it from the folder with the original `.aillyrc` - `ailly python/20_PLAN.md`. |
| 225 | +1. Create `21{a,b,c}_PLAN_DETAILS.md` |
| 226 | +
|
| 227 | + - Write a prompt that instructs the model to write itself prompts: |
| 228 | +
|
| 229 | + ``` |
| 230 | + Based on the plan, create a human prompt for each method in the SES2Mailer class, as well as the main method. |
| 231 | +
|
| 232 | + Format your output as a shell heredoc cat that writes the prompt into a markdown file. |
| 233 | +
|
| 234 | + The markdown should have yaml greymatter with two properties - `skip: true` and `prompt: ` with the content you generate. |
| 235 | +
|
| 236 | + The prompt will be used as the `human` side of an LLM conversation. |
| 237 | +
|
| 238 | + The file should get written to `51_{function_name}.md`. |
| 239 | +
|
| 240 | + Start with these methods: |
| 241 | + - def create_email_identity(self): |
| 242 | + - def create_contact_list(self, contact_list_name): |
| 243 | + [... etc, copied from the plan] |
| 244 | + ``` |
| 245 | +
|
| 246 | + - This is the first "model writing for the model" step. |
| 247 | + - After running, you should get a shell script that will make these files. |
| 248 | + - You can also make the files yourselves |
| 249 | +
|
| 250 | +1. Create and generate the `51_{function name}.md` files from the PLAN_DETAILS step. |
| 251 | +1. Repeat for testing (or did you proactively add testing in the original plan? Nice!) |
| 252 | +1. Consolidate everything you have at this point to a new project. Maybe have Ailly write a script that does it for you? |
| 253 | +
|
| 254 | +## Run and Test? |
| 255 | +
|
| 256 | +1. Create a new file, and paste the generated code in. |
| 257 | +1. Use IDE tooling to fix type issues; run it, test it, etc. |
| 258 | +1. Copy your edits back into the output, and add `skip: true` to lock them in. |
| 259 | +1. Iterate. If your plan had 4 functions, but you wanted 5, edit the plan and rerun the steps after the plan. |
| 260 | +1. If your main function did some special set up, edit it as well. |
| 261 | +1. Edit the "first" function with your patterns and best practices. This will 'level set' future answers for running after this step. |
| 262 | +1. Continue iteratively prompting for the next step of the plan, copying it to the working directory, running, testing, editing, and copying back. |
| 263 | +
|
| 264 | +# Second Language |
| 265 | +
|
| 266 | +1. Copy all your work from the first language to a second language. |
| 267 | +1. Keep `skip: true` on README, Spec, and other language independent files. |
| 268 | +1. Redo the API lookup and extraction. |
| 269 | +1. Replace all first language- prompt words with the second - eg "write python to" with "write java to" |
| 270 | +1. Rerun for the second language. |
| 271 | +1. Re-iterate for the Run and Test steps. |
| 272 | +
|
| 273 | +# Metadata |
| 274 | +
|
| 275 | +1. Provide examples of metadata? |
| 276 | +1. Provide metadata docs and instruction? |
| 277 | +1. Ask for a patch to add snippet tags? |
| 278 | + - This didn't actually work - the patch rarely applies cleanly, and the locations aren't what I'd want. |
| 279 | +
|
| 280 | +# PLAN |
| 281 | +
|
| 282 | +- **Prepare** a precise prompt (by writing an aillyrc system prompt, providing supporting documents, and giving individual prompt steps). |
| 283 | +- **Leverage** LLM models (by running Ailly on some or all parts of the context chain). |
| 284 | +- **Assess** the generated content (as Ailly and the LLM writes output, make sure it's on the right track). |
| 285 | +- **Narrow** your context (by editing Ailly's generated content to keep the conversation going where you want it to). |
0 commit comments