-
Notifications
You must be signed in to change notification settings - Fork 591
/
CHANGES.txt
212 lines (206 loc) · 14.7 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
Release Notes - StreamSets Data Collector - Version 1.1.0
** Sub-task
* [SDC-332] - Support Java Security Manager with strict permissions for user code
* [SDC-1427] - IPC Destination
* [SDC-1479] - HBase should not send records to error on any error
* [SDC-1481] - IPC Origin
* [SDC-1539] - Add @ElDef annotaton for auto discovery of custom EL functions
* [SDC-1540] - Register classes with @ElDef as providers of ELs for all configs in the library
* [SDC-1562] - Replace field path in Record Field with single quote and double quote escaped field path
* [SDC-1586] - Document Delimited record support by name changes
* [SDC-1627] - Show both matched and not matched sample records in Data Summary page
** Bug
* [SDC-1165] - Processor for renaming fields
* [SDC-1166] - Processor for merging list and map fields
* [SDC-1177] - Remove user/pipeline names from log messages, they are auto-injected
* [SDC-1178] - Stage configs should be re-injected into beans
* [SDC-1201] - Fix flaky TestMultiplePipelinesComplex
* [SDC-1237] - JDBC Destination
* [SDC-1250] - Alert on pipeline events e.g. stopped.
* [SDC-1324] - Deleting pipeline not deleting runInfo directory containing state/offset/history/snapshots
* [SDC-1330] - Disable Stop button for STARTING state when pipeline running in cluster mode
* [SDC-1343] - Directory Source doesn't allow buffer limit to be set greater or equal to 1000KB
* [SDC-1349] - Cannot run pipeline after switching the execution mode from cluster to standalone with kafka source
* [SDC-1357] - Pipeline memory consumption exception mentions in bytes instead of MB
* [SDC-1368] - Show source offset in History/View Summary modal dialog similar to Current Source Offset in Summary page
* [SDC-1379] - Duplicate Pipeline State - RUNNING_ERROR and RUN_ERROR.
* [SDC-1390] - Pipeline fails to start in cluster mode when pipeline name contains space
* [SDC-1392] - Validation message is not user friendly, we need to include stage instance label and library label instead of name in validation message
* [SDC-1393] - Move 'No Data Available' label to center in 'Worker SDC Instance' panel similar to other panels
* [SDC-1396] - Snapshot is capturing more than 10 records when pipeline is running Slave mode (Worker)
* [SDC-1399] - Notifications icon (next to Home) and Alerts icon (monitor mode) should have different icons or the same name
* [SDC-1409] - Fix chart rendering in Stage Batch Processing Timer & Stage Heap Memory Usage Pie chart when value is zero
* [SDC-1416] - Need Delimited text support in HDFS Origin
* [SDC-1418] - helpref.properties should be placed in a jar
* [SDC-1429] - Cluster mode prints warnings about offsets
* [SDC-1441] - Field merger should catch IllegalArgumentException for setting fields missing parent.
* [SDC-1447] - No metrics in batch mode when the job is short
* [SDC-1450] - Pipeline canvas shouldn't change view when starting and stopping pipeline
* [SDC-1456] - SDC init.d script may report that process is started, but immediately fail
* [SDC-1458] - KafkaTarget sends records to error on any error
* [SDC-1464] - Clicking on Stop pipeline in UI is currently waiting for some time since we added sleep in the code. Stop pipeline needs to be async operation
* [SDC-1466] - SourceRunner should invoke commit() if the source implements OffsetCommitter
* [SDC-1469] - Binary Data Parser and Data Generator for stages.
* [SDC-1471] - Store only up to 10 minutes of JVM Metrics data in JVM Metrics page
* [SDC-1473] - Callback from slave fails when pipeline is in connect error state
* [SDC-1475] - Remove Stage input/output/error/stage error counter from metric registry
* [SDC-1488] - cluster mode does not work when SDC_CONF is not default
* [SDC-1489] - Add UI Integration tests for Preview
* [SDC-1493] - Remove Pipeline Name from URL when updating page view in Google Analytics
* [SDC-1499] - Hadoop origin should throw validation error if config dir is absolute
* [SDC-1505] - Clicking on home page should refresh pipelines list instead of showing stale information
* [SDC-1506] - Validate a ConfigDefBean is not a primitive type
* [SDC-1507] - Remove unused property value() from ComplexField annotation
* [SDC-1508] - Set minimum heap equal to maximum heap in SDC_JAVA_OPTS
* [SDC-1509] - ConfigDefBean should support group mapping
* [SDC-1512] - Cannot use Snappy in multiple Hadoop Targets per SDC
* [SDC-1529] - Secure SDC must renew tickets before expiration
* [SDC-1533] - Missing fields in HBase Target cause NullPointerException
* [SDC-1542] - When stopping Flume pipeline, will wait for timeout of 2minutes by default
* [SDC-1543] - HBase Target should set autoFlush = false
* [SDC-1547] - Rename and clean up the IPC origin
* [SDC-1550] - Rename and clean up the IPC origin and destination
* [SDC-1552] - Deleting pipeline doesn't delete state history, offset in runInfo
* [SDC-1554] - Provide better error message when JSON file should be processed as Array of Records
* [SDC-1558] - Dev Random Error: Allow configuration of output or remove Vanish capability
* [SDC-1563] - Directory origin shuts down pipeline with FINISHED state if batch wait time expires with no records in batch
* [SDC-1568] - Not fetching fields when selecting text or dragging in codemirror widget
* [SDC-1571] - Cluster Pipeline Preview should convert input to text
* [SDC-1573] - SDC log4j default values should be determined by build profiles
* [SDC-1576] - More renames of IPC origin
* [SDC-1578] - RuntimeStatsGauge does not properly expose metrics via JMX
* [SDC-1581] - java.lang.ClassCastException when using Selector Processor
* [SDC-1582] - Clean up JMS Consumer origin & add useful tooltips where possible
* [SDC-1584] - Pipeline metrics and Data Rules data is lost when pipeline is restarted using retry functionality
* [SDC-1587] - Document Preview Config changes (add show header and show type configuration)
* [SDC-1588] - String EL functions are repeated twice for Selector Process Lane Predicate Configration
* [SDC-1589] - Update Pipeline Config Upgrader for 1.1.0
* [SDC-1590] - NEW YARN state is not handled
* [SDC-1592] - List Map Type Clone not working properly
* [SDC-1595] - Fix tests in build
* [SDC-1598] - Add Raw Data Source in dev library for debug purpose
* [SDC-1599] - Cannot use Lz4 in multiple Hadoop Targets per SDC
* [SDC-1605] - S3 origin must support NONE as a post-processing option
* [SDC-1606] - S3 origin needs to support recursive directory traversal
* [SDC-1607] - S3 origin must support filename patterns
* [SDC-1608] - JDBC Destination should support custom parameters for INSERT query
* [SDC-1609] - Stack trace failure from JDBC destination when writing a record with invalid date format
* [SDC-1614] - Ignore TestKafkaTargetSinglePartition
* [SDC-1616] - Support post processing "NONE" for AmazonS3
* [SDC-1619] - when creating resources tarball filter out unreadable files.
* [SDC-1623] - Expose Error handling options [NONE and ARCHIVE] in S3
* [SDC-1628] - Not able to stop pipeline when Pipeline in in RETRY state
* [SDC-1629] - output tar.gz file should be renamed .tgz
* [SDC-1632] - Error with delimited pipleine
* [SDC-1633] - Unable to enter email ID to get notifications on pipeline state change
* [SDC-1635] - JDBC driver not loaded from META-INF/services anymore
* [SDC-1637] - Update LANE_PREDICATE_MAPPING & COMPLEX_FIELD enum name
* [SDC-1642] - RPM should require JRE 1.7+
* [SDC-1643] - Cluster HDFS Pipelines require upgrade
* [SDC-1645] - Fix field selector loading icon position
* [SDC-1648] - Pipeline retries on OnRecordError-> StopPipeline
* [SDC-1649] - Change the default value of delimiter record type to List-Map
* [SDC-1652] - Double quoted field path list doesn't include list type path access
* [SDC-1653] - Missing pipeline and stages configs should be added with defaults on validation
* [SDC-1654] - Preview Table View is sorting filed path for List-Map data type
* [SDC-1665] - Move API to separate git repo
* [SDC-1673] - Add Apache license headers to all UI files
* [SDC-1675] - Exclude .idea at any level
* [SDC-1683] - Jenkins .m2 directory is stored in the workspace and should be excluded from rat
* [SDC-1684] - Update NOTICE/LICENSE/README files
* [SDC-1685] - Update NOTICE/LICENSE/README files
* [SDC-1686] - Replace findbugs dependency with jsr305
* [SDC-1690] - Kafka Avro goes OOM in Cluster mode
* [SDC-1691] - Dockerfile is looking for tar file not tgz file
* [SDC-1693] - Clarify/correct binary label & tooltip
* [SDC-1694] - Handle non-standard MySQL data error codes
* [SDC-1698] - RPC destination [and error target] must validate the connection and make sure the other side is RPC pipeline
* [SDC-1700] - Several error message fixes
* [SDC-1701] - Error notification pops up in all open browsers even if the pipeline in error is not in view
* [SDC-1702] - PDF - fix pdf generation
* [SDC-1703] - Add license headers to e2e-tests module
* [SDC-1706] - Remove Binary DataFormat from SpoolDir, JMS and AWS origins
* [SDC-1709] - wrong aliasing for JDBC destination in sdc.properties
* [SDC-1716] - RPC origin should not support Cluster mode
* [SDC-1720] - Custom ELs cannot be used from Expression Evaluator
* [SDC-1721] - LoginServlet adding password as part of URL when redirecting to j_security_check
* [SDC-1723] - Amazon S3 origin, when configured to read Avro data, throws UnsupportedOperationException
* [SDC-1726] - Fix helpre.properties for all updated stage versions
* [SDC-1729] - Update hosted help link to 1.1.0
* [SDC-1732] - Add basic README to archetype
* [SDC-1733] - Allow users to choose default kafka partitioner and specify partition key in Kafka Target to
* [SDC-1734] - Fix compilation error in 1.0.0b2 branch
* [SDC-1736] - 1.1.0 - Allow users to choose default kafka partitioner and specify partition key in Kafka Target
* [SDC-1737] - Fix AmazonS3 labels
* [SDC-1738] - Update swagger version to 1.5.3 and swagger ui libraries to fix issues in the generated swagger.json
* [SDC-1739] - Remove non-existent github-banner from API README
* [SDC-1742] - MongoDB origin stops when collection has no data
* [SDC-1743] - S3 origin doesn't process unread records from the same file after restart
** New Feature
* [SDC-906] - Hive Destination using Hive Streaming API
* [SDC-1064] - Stages should be able to migrate themselves to its latest config versions
* [SDC-1282] - Implement a SDC IPC Origin and Destination
* [SDC-1291] - JMS Origin
* [SDC-1292] - Add filter by pipeline name & log level support in Log page
* [SDC-1314] - TestStandaloneRunner.testDisconnectedPipelinesStartedAgain failed
* [SDC-1369] - Maven Archetype for building user stage libraries
* [SDC-1372] - Show/Hide record header information in Preview/Snapshot view
* [SDC-1376] - Fail Tail producing more than 10 records in preview when batchSize passed to REST API is 10
* [SDC-1459] - Hortonworks HDFS cluster mode support
* [SDC-1521] - S3 Spooling Origin
* [SDC-1536] - White list for loading stagelibs
* [SDC-1538] - Auto registration of custom EL fuctions
* [SDC-1548] - Add shadow stagelibrary classpath dir outside of installation
* [SDC-1549] - cluster JAR packaging should handle libs commonlib and shadow libs
* [SDC-1580] - Add shield support to elasticsearch
* [SDC-1677] - Move annotation processor to api project
** Task
* [SDC-916] - Add Elasticsearch 1.6 and 1.7 stage libraries.
* [SDC-1329] - Bump up master version to 1.1.0-SNAPSHOT
* [SDC-1465] - Create README/CONTRIB/BUILD .md files
* [SDC-1470] - Fix pipeline memory monitoring
* [SDC-1484] - Rename setDoneFlag to postDestroy
* [SDC-1517] - Convert to uber tarball
* [SDC-1519] - Create libs common-lib to be able to drop EL extension JARs for all stagelibs
* [SDC-1622] - RPC SSL Labels
* [SDC-1624] - Notify on Pipeline State Changes: Add RETRY state
* [SDC-1625] - Pipeline Retry - clean up labels & tooltips
* [SDC-1650] - Make api module a standalone maven project
* [SDC-1656] - Deadlock in SDC due to memory monitoring
* [SDC-1658] - StageIssues exist after successful validation
* [SDC-1660] - sdc init script does not always start sdc on restart
* [SDC-1662] - Udpate license headers to be Apache License
* [SDC-1666] - Delete API from datacollector repo
* [SDC-1667] - Generate 3rd party dependencies report
* [SDC-1676] - Move JDBC Stages to JDBC lib
* [SDC-1678] - Remove annotation processor from sdk and delete sdk-test
* [SDC-1688] - Create 1.1.0-RC1
* [SDC-1717] - Add placeholders for new stages in the helpref.properties
* [SDC-1719] - Create 1.1.0-RC2
* [SDC-1753] - Create 1.1.0 release
** Improvement
* [SDC-144] - Create a Black List / White List for Kafka Topic Origins and Destinations
* [SDC-973] - Better error messages for misuse of the every() function
* [SDC-997] - Importing a new pipeline into sdc is not intuitive
* [SDC-1373] - Automatically mark all non-JVM classes as application classes
* [SDC-1432] - Cluster mode only supports text
* [SDC-1483] - Support infinite scrolling in Log Page instead of clicking load previous log every time
* [SDC-1510] - Improve the CONTAINER_0205 message
* [SDC-1535] - Convert to uber RPM
* [SDC-1541] - Reduce noisy logging in ActiveRecordWriters
* [SDC-1565] - Support loading email templates from JAR
* [SDC-1566] - Field Hasher - Clarify On Field Issue property & option labels
* [SDC-1569] - Make stage white listing backwards compatible
* [SDC-1574] - Add an API in Stage Context to check if the pipeline was stopped
* [SDC-1583] - Move validation and related variables into config beans
* [SDC-1594] - SDCClassLoader should always put protolib JARs first in the classpath
* [SDC-1597] - AmazonS3 related improvements
* [SDC-1600] - Move JMS source to separate stage library
* [SDC-1602] - Support different query types in JDBC Destination for CDC.
* [SDC-1603] - Allow default values outside of ConfigDef
* [SDC-1634] - Rename model annotations
* [SDC-1641] - IPC source and destination should support compression
* [SDC-1668] - Allow OnRecordErrorException to store a reference to a record.
* [SDC-1682] - Notify on DISCONNECTED and CONNECTING states, remove notification on RETRY state
* [SDC-1695] - Data Collector configuration should support references to ENV vars
* [SDC-1710] - Add DO NOT CHANGE message to aliasing configs