Merge remote-tracking branch 'upstream/master' into unsigned64bits_integer

Author: mayya-sharipova

.ci/bwcVersions

@@ -22,5 +22,6 @@ BWC_VERSION:
   - "7.8.0"
   - "7.8.1"
   - "7.9.0"
+  - "7.9.1"
   - "7.10.0"
   - "8.0.0"

.ci/os.sh

@@ -31,12 +31,6 @@ cp -v .ci/init.gradle $HOME/.gradle/init.d
 
 unset JAVA_HOME
 
-if ! [ -e "/usr/bin/bats" ] ; then
-  git clone https://github.com/sstephenson/bats /tmp/bats
-  sudo /tmp/bats/install.sh /usr
-fi
-
-
 if [ -f "/etc/os-release" ] ; then
     cat /etc/os-release
     . /etc/os-release
@@ -54,16 +48,8 @@ else
 fi
 
 sudo bash -c 'cat > /etc/sudoers.d/elasticsearch_vars'  << SUDOERS_VARS
-    Defaults   env_keep += "ZIP"
-    Defaults   env_keep += "TAR"
-    Defaults   env_keep += "RPM"
-    Defaults   env_keep += "DEB"
-    Defaults   env_keep += "PACKAGING_ARCHIVES"
-    Defaults   env_keep += "PACKAGING_TESTS"
-    Defaults   env_keep += "BATS_UTILS"
-    Defaults   env_keep += "BATS_TESTS"
-    Defaults   env_keep += "SYSTEM_JAVA_HOME"
     Defaults   env_keep += "JAVA_HOME"
+    Defaults   env_keep += "SYSTEM_JAVA_HOME"
 SUDOERS_VARS
 sudo chmod 0440 /etc/sudoers.d/elasticsearch_vars
 

CONTRIBUTING.md

@@ -55,6 +55,18 @@ You will need to fork the main Elasticsearch code or documentation repository an
 
 Further instructions for specific projects are given below.
 
+### Tips for code changes
+Following these tips prior to raising a pull request will speed up the review
+cycle.
+
+* Add appropriate unit tests (details on writing tests can be found in the
+  [TESTING](TESTING.asciidoc) file)
+* Add integration tests, if applicable
+* Make sure the code you add follows the [formatting guidelines](#java-language-formatting-guidelines)
+* Lines that are not part of your change should not be edited (e.g. don't format
+  unchanged lines, don't reorder existing imports)
+* Add the appropriate [license headers](#license-headers) to any new files
+
 ### Submitting your changes
 
 Once your changes and tests are ready to submit for review:

README.asciidoc

@@ -35,154 +35,148 @@ First of all, DON'T PANIC. It will take 5 minutes to get the gist of what Elasti
 
 * https://www.elastic.co/downloads/elasticsearch[Download] and unpack the Elasticsearch official distribution.
 * Run `bin/elasticsearch` on Linux or macOS. Run `bin\elasticsearch.bat` on Windows.
-* Run `curl -X GET http://localhost:9200/`.
-* Start more servers ...
+* Run `curl -X GET http://localhost:9200/` to verify Elasticsearch is running.
 
 === Indexing
 
-Let's try and index some twitter like information. First, let's index some tweets (the `twitter` index will be created automatically):
+First, index some sample JSON documents. The first request automatically creates
+the `my-index-000001` index.
 
 ----
-curl -XPUT 'http://localhost:9200/twitter/_doc/1?pretty' -H 'Content-Type: application/json' -d '
+curl -X POST 'http://localhost:9200/my-index-000001/_doc?pretty' -H 'Content-Type: application/json' -d '
 {
-  "user": "kimchy",
-  "post_date": "2009-11-15T13:12:00",
-  "message": "Trying out Elasticsearch, so far so good?"
+  "@timestamp": "2099-11-15T13:12:00",
+  "message": "GET /search HTTP/1.1 200 1070000",
+  "user": {
+    "id": "kimchy"
+  }
 }'
 
-curl -XPUT 'http://localhost:9200/twitter/_doc/2?pretty' -H 'Content-Type: application/json' -d '
+curl -X POST 'http://localhost:9200/my-index-000001/_doc?pretty' -H 'Content-Type: application/json' -d '
 {
-  "user": "kimchy",
-  "post_date": "2009-11-15T14:12:12",
-  "message": "Another tweet, will it be indexed?"
+  "@timestamp": "2099-11-15T14:12:12",
+  "message": "GET /search HTTP/1.1 200 1070000",
+  "user": {
+    "id": "elkbee"
+  }
 }'
 
-curl -XPUT 'http://localhost:9200/twitter/_doc/3?pretty' -H 'Content-Type: application/json' -d '
+curl -X POST 'http://localhost:9200/my-index-000001/_doc?pretty' -H 'Content-Type: application/json' -d '
 {
-  "user": "elastic",
-  "post_date": "2010-01-15T01:46:38",
-  "message": "Building the site, should be kewl"
+  "@timestamp": "2099-11-15T01:46:38",
+  "message": "GET /search HTTP/1.1 200 1070000",
+  "user": {
+    "id": "elkbee"
+  }
 }'
 ----
 
-Now, let's see if the information was added by GETting it:
+=== Search
 
-----
-curl -XGET 'http://localhost:9200/twitter/_doc/1?pretty=true'
-curl -XGET 'http://localhost:9200/twitter/_doc/2?pretty=true'
-curl -XGET 'http://localhost:9200/twitter/_doc/3?pretty=true'
-----
-
-=== Searching
-
-Mmm search..., shouldn't it be elastic?
-Let's find all the tweets that `kimchy` posted:
+Next, use a search request to find any documents with a `user.id` of `kimchy`.
 
 ----
-curl -XGET 'http://localhost:9200/twitter/_search?q=user:kimchy&pretty=true'
+curl -X GET 'http://localhost:9200/my-index-000001/_search?q=user.id:kimchy&pretty=true'
 ----
 
-We can also use the JSON query language Elasticsearch provides instead of a query string:
+Instead of a query string, you can use Elasticsearch's
+https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html[Query
+DSL] in the request body.
 
 ----
-curl -XGET 'http://localhost:9200/twitter/_search?pretty=true' -H 'Content-Type: application/json' -d '
+curl -X GET 'http://localhost:9200/my-index-000001/_search?pretty=true' -H 'Content-Type: application/json' -d '
 {
   "query" : {
-    "match" : { "user": "kimchy" }
+    "match" : { "user.id": "kimchy" }
   }
 }'
 ----
 
-Just for kicks, let's get all the documents stored (we should see the tweet from `elastic` as well):
+You can also retrieve all documents in `my-index-000001`.
 
 ----
-curl -XGET 'http://localhost:9200/twitter/_search?pretty=true' -H 'Content-Type: application/json' -d '
+curl -X GET 'http://localhost:9200/my-index-000001/_search?pretty=true' -H 'Content-Type: application/json' -d '
 {
   "query" : {
     "match_all" : {}
   }
 }'
 ----
 
-We can also do range search (the `post_date` was automatically identified as date)
+During indexing, Elasticsearch automatically mapped the `@timestamp` field as a
+date. This lets you run a range search.
 
 ----
-curl -XGET 'http://localhost:9200/twitter/_search?pretty=true' -H 'Content-Type: application/json' -d '
+curl -X GET 'http://localhost:9200/my-index-000001/_search?pretty=true' -H 'Content-Type: application/json' -d '
 {
   "query" : {
     "range" : {
-      "post_date" : { "from" : "2009-11-15T13:00:00", "to" : "2009-11-15T14:00:00" }
+      "@timestamp": {
+        "from": "2099-11-15T13:00:00",
+        "to": "2099-11-15T14:00:00"
+      }
     }
   }
 }'
 ----
 
-There are many more options to perform search, after all, it's a search product no? All the familiar Lucene queries are available through the JSON query language, or through the query parser.
-
-=== Multi Tenant and Indices
-
-Man, that twitter index might get big (in this case, index size == valuation). Let's see if we can structure our twitter system a bit differently in order to support such large amounts of data.
+=== Multiple indices
 
-Elasticsearch supports multiple indices. In the previous example we used an index called `twitter` that stored tweets for every user.
+Elasticsearch supports multiple indices. The previous examples used an index
+called `my-index-000001`. You can create another index, `my-index-000002`, to
+store additional data when `my-index-000001` reaches a certain age or size. You
+can also use separate indices to store different types of data.
 
-Another way to define our simple twitter system is to have a different index per user (note, though that each index has an overhead). Here is the indexing curl's in this case:
+You can configure each index differently. The following request
+creates `my-index-000002` with two primary shards rather than the default of
+one. This may be helpful for larger indices.
 
 ----
-curl -XPUT 'http://localhost:9200/kimchy/_doc/1?pretty' -H 'Content-Type: application/json' -d '
+curl -X PUT 'http://localhost:9200/my-index-000002?pretty' -H 'Content-Type: application/json' -d '
 {
-  "user": "kimchy",
-  "post_date": "2009-11-15T13:12:00",
-  "message": "Trying out Elasticsearch, so far so good?"
-}'
-
-curl -XPUT 'http://localhost:9200/kimchy/_doc/2?pretty' -H 'Content-Type: application/json' -d '
-{
-  "user": "kimchy",
-  "post_date": "2009-11-15T14:12:12",
-  "message": "Another tweet, will it be indexed?"
+  "settings" : {
+    "index.number_of_shards" : 2
+  }
 }'
 ----
 
-The above will index information into the `kimchy` index. Each user will get their own special index.
-
-Complete control on the index level is allowed. As an example, in the above case, we might want to change from the default 1 shard with 1 replica per index, to 2 shards with 1 replica per index (because this user tweets a lot). Here is how this can be done (the configuration can be in yaml as well):
+You can then add a document to `my-index-000002`.
 
 ----
-curl -XPUT http://localhost:9200/another_user?pretty -H 'Content-Type: application/json' -d '
+curl -X POST 'http://localhost:9200/my-index-000002/_doc?pretty' -H 'Content-Type: application/json' -d '
 {
-  "settings" : {
-    "index.number_of_shards" : 2,
-    "index.number_of_replicas" : 1
+  "@timestamp": "2099-11-16T13:12:00",
+  "message": "GET /search HTTP/1.1 200 1070000",
+  "user": {
+    "id": "kimchy"
   }
 }'
 ----
 
-Search (and similar operations) are multi index aware. This means that we can easily search on more than one
-index (twitter user), for example:
+You can search and perform other operations on multiple indices with a single
+request. The following request searches `my-index-000001` and `my-index-000002`.
 
 ----
-curl -XGET 'http://localhost:9200/kimchy,another_user/_search?pretty=true' -H 'Content-Type: application/json' -d '
+curl -X GET 'http://localhost:9200/my-index-000001,my-index-000002/_search?pretty=true' -H 'Content-Type: application/json' -d '
 {
   "query" : {
     "match_all" : {}
   }
 }'
 ----
 
-Or on all the indices:
+You can omit the index from the request path to search all indices.
 
 ----
-curl -XGET 'http://localhost:9200/_search?pretty=true' -H 'Content-Type: application/json' -d '
+curl -X GET 'http://localhost:9200/_search?pretty=true' -H 'Content-Type: application/json' -d '
 {
   "query" : {
     "match_all" : {}
   }
 }'
 ----
 
-And the cool part about that? You can easily search on multiple twitter users (indices), with different boost levels per user (index), making social search so much simpler (results from my friends rank higher than results from friends of my friends).
-
-=== Distributed, Highly Available
+=== Distributed, highly available
 
 Let's face it, things will fail....
 
@@ -194,7 +188,7 @@ In order to play with the distributed nature of Elasticsearch, simply bring more
 
 We have just covered a very small portion of what Elasticsearch is all about. For more information, please refer to the https://www.elastic.co/products/elasticsearch[elastic.co] website. General questions can be asked on the https://discuss.elastic.co[Elastic Forum] or https://ela.st/slack[on Slack]. The Elasticsearch GitHub repository is reserved for bug reports and feature requests only.
 
-=== Building from Source
+=== Building from source
 
 Elasticsearch uses https://gradle.org[Gradle] for its build system.
 

Vagrantfile

@@ -333,7 +333,7 @@ def sles_common(config, name)
     zypper ar http://download.opensuse.org/distribution/12.3/repo/oss/ oss
     zypper --non-interactive  --gpg-auto-import-keys refresh
     zypper --non-interactive install git-core
-    # choose to "ignore some dependencies" of expect, which has a problem with tcl... 
+    # choose to "ignore some dependencies" of expect, which has a problem with tcl...
     zypper --non-interactive install --force-resolution expect
   SHELL
   suse_common config, name, extra: extra
@@ -465,38 +465,13 @@ def sh_install_deps(config,
 
     #{extra}
 
-    installed java || {
-      echo "==> Java is not installed"
-      return 1
-    }
-    cat \<\<JAVA > /etc/profile.d/java_home.sh
-if [ ! -z "\\\$JAVA_HOME" ]; then
-  export SYSTEM_JAVA_HOME=\\\$JAVA_HOME
-  unset JAVA_HOME
-fi
-JAVA
     ensure tar
     ensure curl
     ensure unzip
     ensure rsync
     ensure expect
 
-    installed bats || {
-      # Bats lives in a git repository....
-      ensure git
-      echo "==> Installing bats"
-      git clone https://github.com/sstephenson/bats /tmp/bats
-      # Centos doesn't add /usr/local/bin to the path....
-      /tmp/bats/install.sh /usr
-      rm -rf /tmp/bats
-    }
-
     cat \<\<SUDOERS_VARS > /etc/sudoers.d/elasticsearch_vars
-Defaults   env_keep += "BATS_UTILS"
-Defaults   env_keep += "BATS_TESTS"
-Defaults   env_keep += "BATS_PLUGINS"
-Defaults   env_keep += "BATS_UPGRADE"
-Defaults   env_keep += "PACKAGE_NAME"
 Defaults   env_keep += "JAVA_HOME"
 Defaults   env_keep += "SYSTEM_JAVA_HOME"
 SUDOERS_VARS
@@ -505,21 +480,9 @@ SUDOERS_VARS
 end
 
 def windows_common(config, name)
-  config.vm.provision 'markerfile', type: 'shell', inline: <<-SHELL
-    $ErrorActionPreference = "Stop"
-    New-Item C:/is_vagrant_vm -ItemType file -Force | Out-Null
-  SHELL
-
   config.vm.provision 'set prompt', type: 'shell', inline: <<-SHELL
     $ErrorActionPreference = "Stop"
     $ps_prompt = 'function Prompt { "#{name}:$($ExecutionContext.SessionState.Path.CurrentLocation)>" }'
     $ps_prompt | Out-File $PsHome/Microsoft.PowerShell_profile.ps1
   SHELL
-
-  config.vm.provision 'set env variables', type: 'shell', inline: <<-SHELL
-    $ErrorActionPreference = "Stop"
-    [Environment]::SetEnvironmentVariable("PACKAGING_ARCHIVES", "C:/project/build/packaging/archives", "Machine")
-    [Environment]::SetEnvironmentVariable("PACKAGING_TESTS", "C:/project/build/packaging/tests", "Machine")
-    [Environment]::SetEnvironmentVariable("JAVA_HOME", $null, "Machine")
-  SHELL
 end

benchmarks/README.md

@@ -63,3 +63,29 @@ To get realistic results, you should exercise care when running benchmarks. Here
 * Blindly believe the numbers that your microbenchmark produces but verify them by measuring e.g. with `-prof perfasm`.
 * Run more threads than your number of CPU cores (in case you run multi-threaded microbenchmarks).
 * Look only at the `Score` column and ignore `Error`. Instead take countermeasures to keep `Error` low / variance explainable.
+
+## Disassembling
+
+Disassembling is fun! Maybe not always useful, but always fun! Generally, you'll want to install `perf` and FCML's `hsdis`.
+`perf` is generally available via `apg-get install perf` or `pacman -S perf`. FCML is a little more involved. This worked
+on 2020-08-01:
+
+```
+wget https://github.com/swojtasiak/fcml-lib/releases/download/v1.2.2/fcml-1.2.2.tar.gz
+tar xf fcml*
+cd fcml*
+./configure
+make
+cd example/hsdis
+make
+cp .libs/libhsdis.so.0.0.0
+sudo cp .libs/libhsdis.so.0.0.0 /usr/lib/jvm/java-14-adoptopenjdk/lib/hsdis-amd64.so
+```
+
+If you want to disassemble a single method do something like this:
+
+```
+gradlew -p benchmarks run --args ' MemoryStatsBenchmark -jvmArgs "-XX:+UnlockDiagnosticVMOptions -XX:CompileCommand=print,*.yourMethodName -XX:PrintAssemblyOptions=intel"
+```
+
+If you want `perf` to find the hot methods for you then do add `-prof:perfasm`.

buildSrc/build.gradle

@@ -99,7 +99,7 @@ dependencies {
   api 'com.netflix.nebula:gradle-info-plugin:7.1.3'
   api 'org.apache.rat:apache-rat:0.11'
   api "org.elasticsearch:jna:5.5.0"
-  api 'com.github.jengelman.gradle.plugins:shadow:5.1.0'
+  api 'com.github.jengelman.gradle.plugins:shadow:6.0.0'
   api 'de.thetaphi:forbiddenapis:3.0'
   api 'com.avast.gradle:gradle-docker-compose-plugin:0.12.1'
   api 'org.apache.maven:maven-model:3.6.2'