Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion _ml-commons-plugin/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ has_toc: false

ML Commons for OpenSearch eases the development of machine learning features by providing a set of common machine learning (ML) algorithms through transport and REST API calls. Those calls choose the right nodes and resources for each ML request and monitors ML tasks to ensure uptime. This allows you to leverage existing open-source ML algorithms and reduce the effort required to develop new ML features.

Interaction with the ML Commons plugin occurs through either the [REST API]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api) or [AD]({{site.url}}{{site.baseurl}}/observability-plugin/ppl/commands#ad) and [kmeans]({{site.url}}{{site.baseurl}}/observability-plugin/ppl/commands#kmeans) Piped Processing Language (PPL) commands.
Interaction with the ML Commons plugin occurs through either the [REST API]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api) or [AD]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/functions#ad) and [kmeans]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/functions#kmeans) Piped Processing Language (PPL) commands.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think i brought this up previously but couldn't remember the conclusion. SQL is under search category and PPL is under observability category because they have different use cases from user's perspective. is there a reason to move PPL out of observability?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with you -- the use cases are different. I moved to SQL, because it is implemented by SQL plugin. I'm welcome for proposals and ready to move it back.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

got it. I don't have a strong opinion on this, i think it's ok if no objections


Models [trained]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api#train-model) through the ML Commons plugin support model-based algorithms such as kmeans. After you've trained a model enough so that it meets your precision requirements, you can apply the model to [predict]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api#predict) new data safely.

Expand Down
4 changes: 2 additions & 2 deletions _observability-plugin/app-analytics.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ To get started, choose **Observability** in OpenSearch Dashboards, and then choo
2. Enter a name for your application and optionally add a description.
3. Do at least one of the following:

- Use [PPL]({{site.url}}{{site.baseurl}}/observability-plugin/ppl/index) to specify the base query.
- Use [PPL]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index) to specify the base query.

You can't change the base query after the application is created.
{: .note }
Expand All @@ -31,7 +31,7 @@ You can't change the base query after the application is created.
### Create a visualization

1. Choose the **Log Events** tab.
1. Use [PPL]({{site.url}}{{site.baseurl}}/observability-plugin/ppl/index) to build upon your base query.
1. Use [PPL]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index) to build upon your base query.
1. Choose the **Visualizations** tab to see your visualizations.
1. Expand the **Save** dropdown menu, enter a name for your visualization, then choose **Save**.

Expand Down
4 changes: 2 additions & 2 deletions _observability-plugin/event-analytics.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ nav_order: 10

# Event analytics

Event analytics in observability is where you can use [Piped Processing Language]({{site.url}}{{site.baseurl}}/observability-plugin/ppl/index) (PPL) queries to build and view different visualizations of your data.
Event analytics in observability is where you can use [Piped Processing Language]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index) (PPL) queries to build and view different visualizations of your data.

## Get started with event analytics

Expand All @@ -24,7 +24,7 @@ source = opensearch_dashboards_sample_data_logs | fields host | stats count()

By default, Dashboards shows results from the last 15 minutes of your data. To see data from a different timeframe, use the date and time selector.

For more information about building PPL queries, see [Piped Processing Language]({{site.url}}{{site.baseurl}}/observability-plugin/ppl/index).
For more information about building PPL queries, see [Piped Processing Language]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index).

## Save a visualization

Expand Down
3 changes: 1 addition & 2 deletions _observability-plugin/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ nav_order: 1
has_children: false
redirect_from:
- /observability-plugin/
- /observability-plugin/
---

# About Observability
Expand All @@ -16,7 +15,7 @@ Observability is collection of plugins and applications that let you visualize d

Your experience of exploring data might differ, but if you're new to exploring data to create visualizations, we recommend trying a workflow like the following:

1. Explore data over a certain timeframe using [Piped Processing Language]({{site.url}}{{site.baseurl}}/observability-plugin/ppl/index).
1. Explore data over a certain timeframe using [Piped Processing Language]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index).
2. Use [event analytics]({{site.url}}{{site.baseurl}}/observability-plugin/event-analytics) to turn data-driven events into visualizations.
![Sample Event Analytics View]({{site.url}}{{site.baseurl}}/images/event-analytics.png)
3. Create [operational panels]({{site.url}}{{site.baseurl}}/observability-plugin/operational-panels) and add visualizations to compare data the way you like.
Expand Down
2 changes: 1 addition & 1 deletion _observability-plugin/operational-panels.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ nav_order: 30

# Operational panels

Operational panels in OpenSearch Dashboards are collections of visualizations generated using [Piped Processing Language]({{site.url}}{{site.baseurl}}/observability-plugin/ppl/index) (PPL) queries.
Operational panels in OpenSearch Dashboards are collections of visualizations generated using [Piped Processing Language]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index) (PPL) queries.

## Get started with operational panels

Expand Down
10 changes: 10 additions & 0 deletions _observability-plugin/ppl.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
layout: default
title: Piped Processing Language
nav_order: 40
---

# Piped Processing Language

Observability plugin uses Piped Processing Language (`PPL`). Since this language is provided by the **SQL Plugin**, the `PPL` documentation can be found in [SQL Plugin]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index/) section.

36 changes: 0 additions & 36 deletions _observability-plugin/ppl/datatypes.md

This file was deleted.

24 changes: 0 additions & 24 deletions _observability-plugin/ppl/endpoint.md

This file was deleted.

10 changes: 0 additions & 10 deletions _observability-plugin/ppl/functions.md

This file was deleted.

71 changes: 0 additions & 71 deletions _observability-plugin/ppl/protocol.md

This file was deleted.

49 changes: 0 additions & 49 deletions _observability-plugin/ppl/settings.md

This file was deleted.

2 changes: 1 addition & 1 deletion _opensearch/data-streams.md
Original file line number Diff line number Diff line change
Expand Up @@ -262,4 +262,4 @@ You can use wildcards to delete more than one data stream.

We recommend deleting data from a data stream using an ISM policy.

You can also use [asynchronous search]({{site.url}}{{site.baseurl}}/search-plugins/async/index/) and [SQL]({{site.url}}{{site.baseurl}}/search-plugins/sql/index/) and [PPL]({{site.url}}{{site.baseurl}}/observability-plugin/ppl/index/) to query your data stream directly. You can also use the security plugin to define granular permissions on the data stream name.
You can also use [asynchronous search]({{site.url}}{{site.baseurl}}/search-plugins/async/index/) and [SQL]({{site.url}}{{site.baseurl}}/search-plugins/sql/index/) and [PPL]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index/) to query your data stream directly. You can also use the security plugin to define granular permissions on the data stream name.
61 changes: 33 additions & 28 deletions _search-plugins/sql/cli.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,24 @@
---
layout: default
title: SQL CLI
parent: SQL
nav_order: 2
title: SQL & PPL CLI
parent: SQL & PPL
nav_order: 3
---

# SQL CLI
# SQL & PPL CLI

SQL CLI is a stand-alone Python application that you can launch with the `opensearchsql` command.
CLI is a stand-alone Python application that you can launch with the `opensearchsql` command.

Install the SQL plugin to your OpenSearch instance, run the CLI using MacOS or Linux, and connect to any valid OpenSearch end-point.

![SQL CLI]({{site.url}}{{site.baseurl}}/images/cli.gif)

## Features

SQL CLI has the following features:
CLI has the following features:

- Multi-line input
- PPL support
- Autocomplete for SQL syntax and index names
- Syntax highlighting
- Formatted output:
Expand All @@ -33,26 +34,16 @@ SQL CLI has the following features:

Launch your local OpenSearch instance and make sure you have the SQL plugin installed.

To install the SQL CLI:

1. We suggest you install and activate a python3 virtual environment to avoid changing your local environment:
```
pip install virtualenv
virtualenv venv
cd venv
source ./bin/activate
```

2. Install the CLI:
```
1. Install the CLI:
```console
pip3 install opensearchsql
```

The SQL CLI only works with Python 3.
{: .note }

3. To launch the CLI, run:
```
2. To launch the CLI, run:
```console
opensearchsql https://localhost:9200 --username admin --password admin
```
By default, the `opensearchsql` command connects to http://localhost:9200.
Expand All @@ -71,32 +62,46 @@ For a list of all available configurations, see [clirc](https://github.com/opens

## Using the CLI

1. Save the sample [accounts test data](https://github.com/opensearch-project/sql/blob/main/doctest/test_data/accounts.json) file.

1. Index the sample data.
1. Run the CLI tool. If your cluster runs with the default security settings, use the following command:
```console
opensearchsql --username admin --password admin https://localhost:9200
```
curl -H "Content-Type: application/x-ndjson" -POST https://localhost:9200/data/_bulk -u 'admin:admin' --insecure --data-binary "@accounts.json"
If your cluster runs without security, run:
```console
opensearchsql
```

1. Run a sample SQL command:
```
2. Run a sample SQL command:
```sql
SELECT * FROM accounts;
```

By default, you see a maximum output of 200 rows. To show more results, add a `LIMIT` clause with the desired value.

## Using the CLI with PPL

1. Run CLI by specifying query language:
```console
opensearchsql -l ppl <params>
```

2. Execute a PPL query:
```sql
source=accounts | fields firstname, lastname
```

## Query options

Run a single query with the following options:
Run a single query with the following command line options:

- `--help`: Help page for options
- `-q`: Follow by a single query
- `-f`: Specify JDBC or raw format output
- `-v`: Display data vertically
- `-e`: Translate SQL to DSL

## CLI options

- `--help`: Help page for options
- `-l`: Query language option. Available options are `sql` and `ppl`. Default is `sql`
- `-p`: Always use pager to display output
- `--clirc`: Provide path for the configuration file
Loading