Skip to content

Commit 1426741

Browse files
ognjenkaticv1r3ndenniscodesbrianhksRizaFarheen
authored
Merge in recent changes (#1)
* Update README.md * Fix CVEs in 3.16. (conductor-oss#46) * Upgraded ES7 to 7.17.16, alpine to 3.19 * Update ElasticSearch to 7.17.16, some sdk tests are failing. * Server would not run, StackOverflow - 54742540 had this handy fix. * Use the new image in test container. * Update Spring Boot to 3.2.1. One failing test in end to end. * Handle the change in exceptions from Spring Framework. * Update AWS SDK for CVE fix. * Replace generic import with specific class. * Removed dependencies.lock file from projects. No longer used. * prevents orkes-queues from pulling in old version of condcutor (conductor-oss#55) * Documentation link update in Readme (conductor-oss#54) * Update README.md * Revert "Update README.md" This reverts commit fe16da9. * Update README.md * Change dependency group,version (conductor-oss#48) * Remove unnecessary Autowired from Constructors (conductor-oss#36) * Remove unnecessary Autowired from Constructors * Add @Autowired back to property classes --------- Co-authored-by: tiffany jernigan <[email protected]> * upgrade from gradle 7.6.2 to gradle 8.5 (conductor-oss#28) * Fix redundant taskDef in DB (conductor-oss#17) * Add imports file that define auto configure classes supported by spring-boot 2.7 (conductor-oss#60) * update the workflow template * fixes * Update README.md * Left menu URL fix for docs site (conductor-oss#77) Update mkdocs.yml by @RizaFarheen * Update source.md (conductor-oss#75) changed excuted -> executed * Update README.md (conductor-oss#72) againt -> against * Optimise the postgres queries for popping messages (conductor-oss#76) * Feature - use environment variable to set ES6 support mode for Nashorn script evaluator (conductor-oss#88) Use environment variable CONDUCTOR_NASHORN_ES6_ENABLED=true to set ES6 support mode for Nashorn script evaluator. * Fix an issue with the postgres indices as they weren't being used properly (conductor-oss#92) * Upgrade the postgres lib to prevent CVE-2024-1597 (conductor-oss#94) * Enable finer-grained control over the postgres indexing and searching (conductor-oss#93) * Enable the searching of the JSON field in Postgres to be configured * Enable the indexing of tasks to be enabled / disabled * Implement removeWorkflow for postgres index * Enable the migration to run without always recreating an index (conductor-oss#98) * Feature: Use TaskStatusListener.onTaskInProgress() (conductor-oss#58) * apply task-task-status-listener * fix test error * prevent NPE * Fix SpringBoot imports file location error (conductor-oss#89) * Implementation of Postgres PollDataDAO with read and write caching (conductor-oss#82) * Initial implementation of an experimental queue optimisation using Postgres LISTEN/NOTIFY (conductor-oss#87) * Initial implementation of an experimental queue optimisation using Postgres LISTEN/NOTIFY * Use Optional instead of returning null for getSize() * Update connection method for queue listener to make it more thread safe * Only apply the queue notify migrations if configured * Update the postgres docs with all configurable properties (conductor-oss#99) * Fix ElasticSearch health check failure when ES sets password (conductor-oss#59) * Add option to only index workflows on Postgres when their status changes (conductor-oss#83) * Implement postgres lock DAO * add postgres locking mention in docs * rename locking db migration * refactor query - the pg now() function is already UTC * fix test * fix pg query * simplify query * Adding redis username support for clusters (conductor-oss#105) Co-authored-by: Anup Chatterjee <[email protected]> * fix tests * timeunit ms * Update README.md * Workflow and task status notifier Co-authored-by: ToastedTaco <[email protected]> Co-authored-by: BabyBlue0214 <[email protected]> Co-authored-by: Elijah Spicer <[email protected]> * Fix PostgresIndexQueryBuilder null pointer exception (conductor-oss#116) * Throw IllegalArgumentException when query string does not match regex * Unit test for exception * Update README.md (conductor-oss#113) * Add a migration to relax the size constraint on task_index columns (conductor-oss#114) * Fix of exception handler trying to write response to a disconnected client (conductor-oss#109) Why: On Conductor server side, an exception gets raised `Failure in @ExceptionHandler com.netflix.conductor.rest.controllers.ApplicationExceptionMapper#handleAll(HttpServletRequest, Throwable)`. Cause is attempting to write while the client aborted the connection `org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe` What: Added a ClientAbortException handler. Testing done: Local run. Not trivial to emulate the issue in a test. Such a test may rely on timing out the client before server responded. Signed-off-by: Iva Avramova <[email protected]> * Workflow bulk deletion API and action module options (conductor-oss#110) * Initial commit to add bulk workflow deletion * Added an API endpoint for bulk deletion * Add archiveworkflow param * Added bulk delete action to BulkActionModule * Added archive option to bulk delete * Addition of terminateRemove() Functions (conductor-oss#9) * Added an API endpoint for bulk deletion * Fixed WorkflowBulkServiceTest, added test cases for deleteWorkflow an… (conductor-oss#10) * Fixed WorkflowBulkServiceTest, added test cases for deleteWorkflow and terminateRemove, and added WorkflowResourceTest test for single terminateRemove * pass spotlessjava --------- Co-authored-by: JeffP <[email protected]> --------- Co-authored-by: jeffp1 <[email protected]> Co-authored-by: Ayush Thengne <[email protected]> Co-authored-by: Jeff P <[email protected]> Co-authored-by: Doe1111 <[email protected]> * revert WorkflowDef changes * Join task fixes (conductor-oss#117) * fix join completing with errors when a task has failed or some tasks are not terminal yet * add more tests * add more comments to join routine * mark join task synchronous * Revert unrelated changes * Status Notifier Properties Cleanup * fix tests * fix formatting * update the tests for the JOIN task fix * Fix the name of the task index migration * Update README.md * Fix of exception handler trying to write response to a disconnected client Why: On Conductor server side, an exception gets raised `Failure in @ExceptionHandler com.netflix.conductor.rest.controllers.ApplicationExceptionMapper#handleAll(HttpServletRequest, Throwable)`. Cause is attempting to write while the client aborted the connection `org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe`, while the exception was wrapped in `ClosedChannelException`. What: Added a ClosedChannelException handler for such exceptions root caused by ClientAbortException. Testing done: Local run. Not trivial to emulate the issue in a test. Such a test may rely on timing out the client before server responded. Signed-off-by: Iva Avramova <[email protected]> * Fix of exception handler trying to write response to a disconnected client Why: On Conductor server side, an exception gets raised `Failure in @ExceptionHandler com.netflix.conductor.rest.controllers.ApplicationExceptionMapper#handleAll(HttpServletRequest, Throwable)`. Cause is attempting to write while the client aborted the connection `org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe`, while the exception was nested in `ClosedChannelException`. What: Added a ClosedChannelException handler for exception chain containing ClientAbortException. Testing done: Local run. Not trivial to emulate the issue in a test. Such a test may rely on timing out the client before server responded. Signed-off-by: Iva Avramova <[email protected]> * fix: Fix error with headers in webhook * Added logic to remove stale iteration outputs based on keepLastN parameter * Remove unused code block * Remove unused code block * Fixed formatting issues * Updating DoWhileTask keepLastN iteration endIndex * style: Fix style * added tests; generalized the exception Signed-off-by: Iva Avramova <[email protected]> * added tests; generalized the exception Signed-off-by: Iva Avramova <[email protected]> * Why: Ignoring a particular exception based on its stack trace should be rather an alerting concern. What: Reverted ClientAbortedException specific handler. Added test coverage for the existing exception handler. Testing done: local run Signed-off-by: Iva Avramova <[email protected]> * Update README.md * if status-listener error, task not executed * change the point of cancellation * Wrap PGExecutionDAO createTasks in a single transaction * publish roadmap * add link to the roadmap * Create codacy.yml * Delete .github/workflows/codacy.yml * Bump io.micrometer:micrometer-registry-datadog from 1.12.1 to 1.13.0 Bumps [io.micrometer:micrometer-registry-datadog](https://github.com/micrometer-metrics/micrometer) from 1.12.1 to 1.13.0. - [Release notes](https://github.com/micrometer-metrics/micrometer/releases) - [Commits](micrometer-metrics/micrometer@v1.12.1...v1.13.0) --- updated-dependencies: - dependency-name: io.micrometer:micrometer-registry-datadog dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Fix for concurrentExecLimit * Bump com.diffplug.spotless from 5.0.0 to 6.25.0 Bumps com.diffplug.spotless from 5.0.0 to 6.25.0. --- updated-dependencies: - dependency-name: com.diffplug.spotless dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: Iva Avramova <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Viren Baraiya <[email protected]> Co-authored-by: Dennis Caldwell <[email protected]> Co-authored-by: Brian Hawkins <[email protected]> Co-authored-by: Riza Farheen <[email protected]> Co-authored-by: Hoony <[email protected]> Co-authored-by: tiffany jernigan <[email protected]> Co-authored-by: tiffany jernigan <[email protected]> Co-authored-by: HOA PHAN <[email protected]> Co-authored-by: Richard Meng <[email protected]> Co-authored-by: Young-Zen <[email protected]> Co-authored-by: Kunal Kumar <[email protected]> Co-authored-by: Ikko Eltociear Ashimine <[email protected]> Co-authored-by: Ben Pirt <[email protected]> Co-authored-by: Vasiliy Pankov <[email protected]> Co-authored-by: Boney <[email protected]> Co-authored-by: Bert Verstraete <[email protected]> Co-authored-by: Anup Chatterjee <[email protected]> Co-authored-by: Anup Chatterjee <[email protected]> Co-authored-by: CollinDewey <[email protected]> Co-authored-by: ToastedTaco <[email protected]> Co-authored-by: BabyBlue0214 <[email protected]> Co-authored-by: Elijah Spicer <[email protected]> Co-authored-by: omadaan <[email protected]> Co-authored-by: ivakoleva <[email protected]> Co-authored-by: Olivia Bailey <[email protected]> Co-authored-by: jeffp1 <[email protected]> Co-authored-by: Ayush Thengne <[email protected]> Co-authored-by: Jeff P <[email protected]> Co-authored-by: Doe1111 <[email protected]> Co-authored-by: Juan Cruz <[email protected]> Co-authored-by: Srihari Malagi <[email protected]> Co-authored-by: Srihari K. Malagi <[email protected]> Co-authored-by: Martini <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: manan164 <[email protected]>
1 parent 2e309b2 commit 1426741

File tree

149 files changed

+4078
-20091
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

149 files changed

+4078
-20091
lines changed

README.md

+15-25
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
![Conductor](docs/img/logo.svg)
22

3-
[![Github release](https://img.shields.io/github/v/release/Netflix/conductor.svg)](https://GitHub.com/Netflix/conductor/releases)
3+
[![Github release](https://img.shields.io/github/v/release/conductor-oss/conductor.svg)](https://GitHub.com/Netflix/conductor-oss/releases)
44
[![License](https://img.shields.io/github/license/conductor-oss/conductor.svg)](http://www.apache.org/licenses/LICENSE-2.0)
55

66

@@ -28,21 +28,22 @@ Please update your forks to point to this repo. This will ensure your commits a
2828
git remote set-url origin https://github.com/conductor-oss/conductor
2929
```
3030
> [!IMPORTANT]
31-
> **Follow the steps below if you have an active PR againt the Netflix/Conductor repository**
31+
> **Follow the steps below if you have an active PR against the Netflix/Conductor repository**
3232
> 1. Fork **this** repository
3333
> 2. Update your local repository to change the remote to this repository
3434
> 3. Send a PR against the `main` branch
3535
36-
## Releases
37-
The latest version is [![Github release](https://img.shields.io/github/v/release/Netflix/conductor.svg)](https://GitHub.com/Netflix/conductor/releases)
36+
## Conductor OSS Roadmap
37+
[See the roadmap for the Conductor](ROADMAP.md)
3838

39-
The next scheduled release is in Dec 2023.
39+
## Releases
40+
The latest version is [![Github release](https://img.shields.io/github/v/release/conductor-oss/conductor.svg)](https://GitHub.com/conductor-oss/conductor/releases)
4041

4142
## Resources
42-
#### [Slack Community](https://join.slack.com/t/orkes-conductor/shared_invite/zt-xyxqyseb-YZ3hwwAgHJH97bsrYRnSZg)
43-
We have an active [community](https://join.slack.com/t/orkes-conductor/shared_invite/zt-xyxqyseb-YZ3hwwAgHJH97bsrYRnSZg) of Conductor users and contributors on the channel.
44-
#### [Documentation Site](https://orkes.io/content)
45-
[Documentation](https://orkes.io/content) and tutorial on how to use Conductor
43+
#### [Slack Community](https://join.slack.com/t/orkes-conductor/shared_invite/zt-2hmxn0i3n-_W~a9rWMbvMoYmlJo3Y15g)
44+
We have an active [community](https://join.slack.com/t/orkes-conductor/shared_invite/zt-2hmxn0i3n-_W~a9rWMbvMoYmlJo3Y15g) of Conductor users and contributors on the channel.
45+
#### [Documentation Site](https://docs.conductor-oss.org/)
46+
[Documentation](https://docs.conductor-oss.org/) and tutorial on how to use Conductor
4647

4748
[Discussion Forum](https://github.com/conductor-oss/conductor/discussions): Please use the forum for questions and discussing ideas and join the community.
4849

@@ -61,28 +62,17 @@ The runnable server is in server/ module.
6162
Follow the steps below to launch the docker container:
6263

6364
```shell
64-
65-
# Create volumes for persistent stores
66-
# Used to create a persistent volume that will preserve the
67-
docker volume create postgres
68-
docker volume create redis
69-
70-
docker run --init -p 8080:8080 -p 1234:5000 --mount source=redis,target=/redis \
71-
--mount source=postgres,target=/pgdata conductoross/conductor-standalone:3.15.0
72-
```
73-
74-
Navigate to http://localhost:1234 once the container starts to launch UI.
75-
76-
## Docker Containers for production usage
77-
```shell
78-
docker pull conductoross/conductor:3.15.0
65+
docker compose -f docker/docker-compose.yaml up
7966
```
67+
* Navigate to http://localhost:5000 once the container starts to launch UI.
68+
* APIs are accessible at http://localhost:8080
69+
* Swagger Docs:http://localhost:8080/swagger-ui/index.html?configUrl=/api-docs/swagger-config#/
8070

8171

8272
## Database Requirements
8373

8474
* The default persistence used is Redis
85-
* The indexing backend is [Elasticsearch](https://www.elastic.co/) (6.x)
75+
* The indexing backend is [Elasticsearch](https://www.elastic.co/) (7.x)
8676

8777
## Other Requirements
8878
* JDK 17+

ROADMAP.md

+75
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Conductor OSS Roadmap
2+
3+
4+
## New Features
5+
### Type safety for workflow inputs and task input/output through JSON Schema
6+
7+
* Allow type safe workflows and workers with support for JSON schema and protobuf
8+
* Enable scaffolding code generation for workers through schema for workers using CLI tool
9+
10+
### New System Tasks
11+
12+
* Database task to work with relational & no-sql databases
13+
* Polling support for HTTP task
14+
* Operators
15+
* * For..Each with parallel and sequential execution
16+
* * Improved While loop
17+
* * Try..Catch for improved error handling at the task level
18+
19+
### LLM Integrations
20+
Conductor is a perfect platform to build your next LLM powered application or incorporating genAI into your applications.
21+
Enable system tasks for LLM integrations that lets you work with various language models for:
22+
1. Text completion
23+
2. Chat completion with memory
24+
3. Embedding generation
25+
26+
### CLI for Conductor
27+
Allow developers to manage their conductor instance via CLI.
28+
29+
* Manage metadata
30+
* Query and manage workflow executions (terminate, pause, resume, retry)
31+
* Start | Stop manage conductor server
32+
33+
### Support Python as a scripting language for INLINE task
34+
Extend usability of Conductor by allowing lightweight python code as INLINE tasks.
35+
36+
### New APIs for workflow state management
37+
38+
* Synchronous execution of workflows
39+
* update workflow variables
40+
* Update tasks synchronously
41+
42+
## SDKs
43+
44+
* Rust
45+
* Kotlin
46+
* C++
47+
* Ruby
48+
* Swift
49+
* Flutter / Dart
50+
* PHP
51+
52+
### Worker metrics on server
53+
Expose an endpoint on the server that can be used by workers to publish worker specific metrics.
54+
This will allow monitoring metrics for all the workers in a distributed system across the entire system.
55+
56+
## Testing
57+
Infrastructure to make workflows easier to test and debug right from the UI and IDE.
58+
59+
### Workflow Debugger
60+
61+
* Ability to debug your workflows during development just like you would do when you write code
62+
* All functionality of a debugger
63+
* Breakpoints add/remove
64+
* Step to next
65+
* Drop to a certain task that was already executed. (going back in time)
66+
* Ability to inspect, modify, add input / output parameters
67+
* Watch Windows to see values of interesting &nbsp;parameters during execution
68+
* Attaching to a certain WF execution
69+
* Remote Task debugging (with SDK Support).. Enable step by step execution in a task worker from the server
70+
71+
## Maintenance
72+
73+
1. Deprecate support for Elasticsearch 6
74+
2. Update support for newer versions of Elasticsearch
75+
2. Improve/Fix JOIN task performance (less about making it performant and more about just fixing the usability) &nbsp;- Done

amqp/dependencies.lock

-177
This file was deleted.

0 commit comments

Comments
 (0)