sync doc with wiki

Author: HouzuoGuo

doc/Cloud-tips.md

@@ -201,10 +201,32 @@ the AWS x-ray daemon program on the server host by following [AWS X-Ray Daemon g
 All interactions between laitos and AWS generate info-level log messages for diagnosis and inspection.
 
 ## Deploy on Microsoft Azure and Google Compute Engine
+
 Simply copy laitos program and its data onto a Linux virtual machine and start laitos right away. It is often useful to
 use systemd integration to launch laitos automatically upon system boot. All flavours of Linux distributions supported
 by Azure can run laitos.
 
+## Deploy on Google App Engine
+
+Clone the laitos repository and create a sub-directory named
+`gcp_appengine_data`. Place the following files into the sub-directory:
+
+- `daemonName` - A plain-text file of comma-separated daemon names to be started
+  on App Engine. The content is analogous to CLI parameter `-daemons`.
+- `appeng-environment.yaml` - A YAML file of program environment variables,
+  for example: `env_variables:\n LAITOS_INDEX_PAGE: "{...}"`
+- `config.json` - A JSON file for the laitos program configuration.
+  * Alternatively, write the program configuration JSON into the environment
+    variable `LAITOS_CONFIG`.
+- Other program data files such as HTML assets.
+
+Then navigate to the cloned directory (`laitos.git`) and run `gcloud app depoy`.
+
+You can find an example in: https://github.com/HouzuoGuo/laitos/tree/master/gcp_appengine_data.example
+
+Note that, the example config file is named `config.json.example` whereas the
+actual config file must be named `config.json`.
+
 ## Deploy on other cloud providers
 laitos runs on nearly all flavours of Linux system, therefore as long as your cloud provider supports Linux compute
 instance, you can be almost certain that it will run laitos smoothly and well.

doc/Get-started.md

@@ -1,27 +1,29 @@
 # Get started
 
-## Acquire software
-Download the latest [laitos software](https://github.com/HouzuoGuo/laitos/releases).
+## Obtain laitos software
 
-For geekier scenarios, use the latest go compiler to compile the software from source code like so:
+Download the latest laitos software from the [releases page](https://github.com/HouzuoGuo/laitos/releases).
 
-    ~/go > go get github.com/HouzuoGuo/laitos
-    ~/go/src/github.com/HouzuoGuo/laitos > go build
+Alternatively, compile the software manually by cloning this repository and then
+run `go build`.
 
-laitos is an all-in-one solution and does not depend on third party library.
+## Craft your configuration
+
+A configurable laitos component belongs to one of the three categories:
 
-## Prepare configuration
-laitos components go into three categories:
 - Apps - reading news and Emails, make a Tweet, ask about weather, etc.
-- Daemons - web/mail/DNS servers, chat bots, etc. Many daemons offer access to apps, protected with a password.
-- Rich web services - useful web-based utilities hosted by the web server.
+  * Some apps do not require manual configuration and they are pre-enabled.
+- Daemons - web/mail/DNS servers, chat bots, etc. Many daemons are capable of
+  accepting app command input and allow command execution protected by a
+  password.
+- Web services - HTML-based utilities, web-hooks for integration with 3rd party
+  services, etc.
 
-Follow the links in [component list](https://github.com/HouzuoGuo/laitos/wiki/Component-list) to craft your very own
-configuration in [JSON](https://en.wikipedia.org/wiki/JSON).
-Keep in mind - nearly all components require configuration to be useful.
+Follow the links in [component list](https://github.com/HouzuoGuo/laitos/wiki/Component-list)
+to craft your very own configuration in [JSON](https://en.wikipedia.org/wiki/JSON).
 
-As an example, here we use laitos DNS server for a safer and ad-free web experience at home, and automatically keep
-the laitos server computer up-to-date with latest security patches:
+As an example, here we use laitos DNS server to provide a safer and ad-free web
+experience at home, and enable a couple of web utilities:
 
     {
       "DNSDaemon": {
@@ -30,14 +32,20 @@ the laitos server computer up-to-date with latest security patches:
           "10."
         ]
       },
-      "Maintenance": {
-        "Recipients": [
-          "[email protected]"
-        ]
-      },
+      "HTTPDaemon": {},
+      "HTTPHandlers": {
+        "CommandFormEndpoint": "/cmd",
+        "FileUploadEndpoint": "/upload",
+        "InformationEndpoint": "/info",
+        "LatestRequestsInspectorEndpoint": "/latest_requests",
+        "ProcessExplorerEndpoint": "/proc",
+        "RequestInspectorEndpoint": "/myrequest",
+        "WebProxyEndpoint": "/proxy"
+      }
     }
 
-## Start program
+## Start the program
+
 Assume that latios software is in current directory, run the following command:
 
     sudo ./laitos -config <PATH TO JSON FILE> -daemons <LIST>
@@ -60,25 +68,62 @@ Note that:
   * [`maintenance`](https://github.com/HouzuoGuo/laitos/wiki/%5BDaemon%5D-system-maintenance) - Automated server maintenance and program health report
 - Apps are enabled automatically once they are configured in the JSON file. Some apps such as the RSS News Reader are automatically enabled via their built-in default configuration.
 
+## Other deployment techniques
+
+### Use environment variables to feed the program configuration
+
+For ease of deployment, laitos can fetch its program configuration along with
+the content of HTTP daemon index page from environment variables - instead of
+the usual files.
+
+When `LAITOS_CONFIG` environment variable is present and not empty, laitos
+program will load its configuration from there. When `LAITOS_INDEX_PAGE`
+environment variable is present and not empty, laitos will use its content
+to serve the index page on its HTTP servers.
+
+Check out environment variable usage examples in the [Kubernetes example](https://github.com/HouzuoGuo/laitos/blob/master/k8s.example/laitos-in-k8s.yaml)
+and the [example in Dockerfile](https://github.com/HouzuoGuo/laitos/blob/master/Dockerfile)
+
+Be aware that the combined size of all environment variables generally cannot
+exceed ~2MBytes.
+
+### Build a container image
+The images of a (usually) up-to-date version of laitos are uploaded to Docker
+Hub [hzgl/laitos](https://hub.docker.com/r/hzgl/laitos).
+
+If you wish to customise the image to your needs, feel free to use the [`Dockerfile`](https://github.com/HouzuoGuo/laitos/blob/master/Dockerfile)
+from GitHub repository as a reference.
+
 ## Deploy on cloud
+
 laitos runs well on all popular cloud vendors, it supports cloud virtual machines for a straight-forward installation,
 as well as more advanced cloud features such as AWS Elastic Beanstalk and AWS Lambda (in combination with API gateway).
 Check out the [cloud deployment tips](https://github.com/HouzuoGuo/laitos/wiki/Cloud-tips).
 
 ## Deploy on Windows
+
 laitos is well tuned for running on Windows server and desktop. Check out this [PowerShell script](https://raw.githubusercontent.com/HouzuoGuo/laitos/master/extra/windows/setup.ps1)
 that helps to start laitos automatically as a background service.
 
-## Advanced behaviours
+## Advanced program behaviours
+
 ### Self-healing
-laitos is extremely reliable thanks to its many built-in mechanisms that activate automatically in the unlikely event of program anormaly.
-The mechanisms are fully automatic and do not require manual intervention:
 
-1. Access to external resources, such as API services on the public Internet, automatically recover from transient errors.
-2. Each daemon automatically restarts in case of a transient initialisation error, such as when required system resource is unavailable.
-3. laitos program and its daemons restart automatically in case of a complete program crash.
-4. In the extremely unlikely event of repeated program crashes in short succession (20 minutes), laitos will restart automatically while
-   shedding of its daemons starting from the heaviest daemon, thus ensuring the maximum availabily of remaining healthy daemons.
+laitos is extremely reliable thanks to its many built-in mechanisms that make
+automated attempts to restart and isolate faulty components. The built-in
+mechanisms are fully automatic and do not require intervention:
+
+1. Automatically recover from transient errors when contacting external
+   resources, such as API services on the public Internet.
+2. Every daemon automatically restarts in case of a transient initialisation
+   error.
+3. In the unlikely event of a program crash, the laitos program automatically
+   restarts itself to recover.
+4. In the extremely unlikely event of repeated program crashes in short
+   succession (20 minutes), laitos will attempt to automatically isolate the
+   faulty daemon by removing daemons before the next restart - shedding the
+   heavier daemons (e.g. DNS) first before shedding the lighter daemons (e.g.
+   HTTP daemon).
 
 Optionally, laitos can send server owner a notification mail when a program crash occurs. To enable the notification, follow
 [outgoing mail configuration](https://github.com/HouzuoGuo/laitos/wiki/Outgoing-mail-configuration) and then specify Email recipients in
@@ -98,6 +143,7 @@ Please use [Github issues](https://github.com/HouzuoGuo/laitos/issues) to report
 output contain valuable clues for diagnosis - please retain them for an issue report.
 
 ### More command line options
+
 Use the following command line options with extra care:
 <table>
 <tr>
@@ -166,19 +212,3 @@ Use the following command line options with extra care:
     </td>
 </tr>
 </table>
-
-### Build a container image
-Images of a (usually) up-to-date version of laitos are uploaded to Docker Hub [hzgl/laitos](https://hub.docker.com/r/hzgl/laitos).
-
-If you wish to customise the image to your needs, feel free to use the [`Dockerfile`](https://github.com/HouzuoGuo/laitos/blob/master/Dockerfile)
-from GitHub repository as a reference.
-
-### Supply program configuration in an environment variable
-Usually, the program configuration is kept in a JSON file, the path of which is specified in the laitos launch command line (`-config my-laitos-config.json`).
-However, if the program configuration is short enough to fit into an environment variable, laitos can also get its configuration
-from there. This can be rather useful for testing a configuration snippet or starting a small number of daemons in a container.
-
-The following example starts the HTTP daemon (without TLS) on the default port number (80), the web server comes with app-command runner endpoint:
-
-    sudo env 'LAITOS_CONFIG={"HTTPFilters": {"PINAndShortcuts": {"Passwords": ["abcdefgh"]},"LintText": {"MaxLength": 1000}},"HTTPHandlers": {"AppCommandEndpoint": "/cmd"}}' ./laitos -daemons insecurehttpd
-    # Try running an app command: curl http://0.0.0.0:80/cmd?cmd=abcdefgh.sdate

doc/[App]-inspect-and-control-server-environment.md

@@ -1,42 +1,60 @@
 ## Introduction
-Check server system health (such as memory usage), and issue controlling commands to the laitos program,
-such as locking and stopping the server daemons.
+
+The app helps to inspect program and system status, including program logs,
+resource usage, and system load.
+
+When invoked with certain action names, the app offers limited control over the
+life-cycle of the program itself (e.g. put it into an unrecoverable locked
+state).
 
 ## Configuration
+
 This app is always available for use and does not require configuration.
 
 ## Usage
-Use any capable laitos daemon to invoke the app:
+
+Use any laitos daemon capable of executing app commands to invoke the app:
 
     .e <action>
 
-Where action can be:
-- `info` - Get program status such as current clock, memory usage, load, etc.
-- `log` - Get latest log entries of all kinds - information and warnings.
-- `warn` - Get latest warning log entries.
-- `stack` - Get the latest stack traces.
+These actions help to inspect the program and system status:
 
-It may also be:
+- `info` - Get a program and system status report including info such as PID,
+  system load, memory usage, etc.
+- `log` - Get the latest log entries of all kinds - information and warnings.
+- `warn` - Get the latest warning log entries.
+- `stack` - Get the latest stack traces.
 - `tune` - Automatically tune server kernel parameters for enhanced performance and security.
-- `lock` - Keep laitos program running, but disable all apps and daemons, All web server URLs will return
-  status 200 (OK) and an error text. The only way to recover from this state is to restart laitos program manually.
-- `stop` - Crash the laitos program.
-- `kill` - Destroy (nearly) all directories and files, mounted and local, on the computer hosting laitos program.
-  Consequently laitos program crashes soon and the host computer will need to be reinitialised.
+
+These actions offer limited control over the life-cycle of the laitos program:
+
+- `lock` - Disable app command execution and disable nearly all daemons with the
+  exception of HTTP servers. Web server handlers will respond with status 200 OK
+  without processing the incoming request.
+  * The only way to recover from this state is to login to the host OS and
+    manually end and then restart the laitos program.
+- `stop` - Cause the laitos program to crash with a panic.
+- `kill` - Erase all disks on the computer host for as long as its OS can
+  endure, which effectively destroys all data on the computer running laitos.
+  * The laitos program will eventually crash along with the host OS.
 
 ## Tips
-- In case that a load balancer periodically checks the health status of laitos by visiting its HTTP server, the checks
-  will continue to succeed (indicating a healthy server) even after `lock` action is executed. This is intentional to
-  prevent the load balancer from replacing the laitos program under lock-down with a healthy instance not under lock-down.
-- The `kill` action attempts to delete most of the files on disk (including those mounted on mount points), and wipes
-  disk partitions with zeros. It cannot guarantee that the entire disk has been filled with zeros before the computer
-  crashes.
-
-About retrieving the log entries:
-- laitos keeps the most recent log entries and warning log entries in memory, totalling several hundreds entries. They
-  are available for inspection on-demand. The host operating system or hosting platform may have held more log entries
-  available for your inspection.
-- The warning log entries keep track of the most recent (about three dozens) repeating offenders and will not present
-  their repeated offences for inspection. For example, when laitos server refuses 30 DNS clients from querying the server
-  yet they keep on going, their IPs will only show up once in the recent warnings, until the server refuses another 30,
-  different DNS clients from querying the server.
+
+- The locked-down mode (after executing the `lock` action) does not stop the
+  HTP servers because HTTP is often used for health check. By responding with
+  status 200 OK the health check will continue to consider laitos program
+  healthy. Without leaving HTTP servers running the health check may decide to
+  restart laitos program which renders the `lock` action ineffective.
+- The `kill` action runs indefinitely for as long as the host OS stays online.
+  It however cannot possibly guarantee that all disks are completely erased
+  before the host OS inevitably crashes.
+
+About the retrieval of log entries:
+
+- laitos keeps the most recent log entries - warnings and information, buffered
+  in memory. Please use the host operating system's facilities
+  (e.g. `journalctl`) to inspect older log entries.
+- To prevent an individual source from spamming the server and generates and
+  thus generate an exceeding quatity of warning logs, the logger will only log a
+  single warning entry for each offender and then suppresses further entries
+  from the same offender for a short period of time.

doc/[Daemon]-mail-server.md

@@ -5,7 +5,6 @@ any mail. For communication secrecy, the server supports StartTLS operation and
 To reduce spam, the sender IPs are passed through these widely-used DNS-based blocklists:
 - "b.barracudacentral.org",
 - "bl.mailspike.net",
-- "bl.spamcop.net",
 - "cbl.abuseat.org",
 - "dnsbl-1.uceprotect.net",
 - "dnsbl-2.uceprotect.net",
@@ -16,7 +15,6 @@ To reduce spam, the sender IPs are passed through these widely-used DNS-based bl
 - "ix.dnsbl.manitu.net",
 - "noptr.spamrats.com",
 - "psbl.surriel.com",
-- "singular.ttk.pte.hu",
 - "spam.dnsbl.anonmails.de",
 - "spam.dnsbl.sorbs.net",
 - "spam.spamrats.com",
@@ -227,17 +225,25 @@ Here are couple of examples involving, assuming that laitos server is on `123.23
 Wait up to an hour for new DNS records to propagate through the Internet.
 
 ## Test
+
 Send a test mail with subject, text, and attachments to any name under `MyDomains` (e.g. `[email protected]`). Wait
 a short moment, check the inbox on any of `ForwardTo` address (e.g. `[email protected]`), the test mail should arrive at
 all of the `ForwardTo` addresses.
 
-Try invoking an app command - send laitos server a mail with arbitrary subject, and write down password PIN and app command
-in the content body. Look for the command response in a mail replied to the sender.
+To invoke an app command, compose a plain text email to laitos server using an
+arbitrary subject text, write down the password PIN and app command in the mail
+body, and send it to laitos server. A short moment later, the command execution
+result will be mailed back to the sender.
 
 ## Tips
+
 - Occasionally your mail provider (such as Gmail) may consider legitimate mails forwarded by laitos as spam, therefore please
   check your spam folders regularly.
 - Many Internet domain names use [DMARC](https://en.wikipedia.org/wiki/DMARC) to protect their business from mail spoofing.
   Though laitos usually forwards the verbatim copy of incoming mail to you, DMARC makes an exception - laitos has to change
   the sender from `[email protected]` to `name@protected-domain-laitos-nodmarc-###.com` where hash is a random digit.
-  Otherwise your mail provder will discard the mail silently - without a trace in spam folder.
+  Otherwise your mail provider will discard the mail silently - without a trace in spam folder.
+- Some mail providers and clients (such as Gmail on the web) automatically
+  attaches a plain-text copy of the rich-text mail content when sending it.
+  When receiving this kind of mail, the laitos mail server will be smart enough
+  to pick up the plain-text copy and look for app command to execute there.