Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MINIFICPP-2146 Add support for SMB networking protocol #1634

Closed
wants to merge 25 commits into from
Closed

MINIFICPP-2146 Add support for SMB networking protocol #1634

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
942ff36
MINIFICPP-2146 Add support for SMB networking protocol
martinzink Jun 23, 2023
d1e8d4c
Update libminifi/src/utils/file/FileUtils.cpp
martinzink Aug 23, 2023
3ab67df
added comment and a test windows file_clock epoch
martinzink Aug 23, 2023
19cde5c
review changes
martinzink Aug 24, 2023
1d5fb29
review changes
martinzink Aug 24, 2023
8978f29
test ListSmb::IgnoreHiddenFiles
martinzink Aug 24, 2023
6dc138d
add /bigobj to WIN32
martinzink Aug 24, 2023
97bf67a
add SMB to ci
martinzink Aug 24, 2023
4c503a0
removed unnecessary newline
martinzink Aug 24, 2023
a4c3787
linter fix
martinzink Aug 25, 2023
64a9325
review changes and rebase fixes
martinzink Sep 5, 2023
7a457d8
add missing documentation
martinzink Sep 8, 2023
03015fc
remove ~PutSmb() override = default;
martinzink Sep 8, 2023
5d06387
ListSmb path attribute to work like NiFi
martinzink Sep 12, 2023
0b8ed3d
fix tests
martinzink Sep 12, 2023
c13cbaa
enable SMB by default
martinzink Sep 12, 2023
0c60601
use count_if
martinzink Sep 12, 2023
8ad193c
erase leading slashes
martinzink Sep 12, 2023
e4e1f1d
use std::to_string instead of fmt::format with simple {}
martinzink Sep 12, 2023
6f5a832
add Put/Fetch/ListFile recommendation in case of automounted SMB shared
martinzink Sep 12, 2023
8f86d3d
only warn in case of missing PutFile::ConflictResolution
martinzink Sep 13, 2023
4dbb2fd
linter fix
martinzink Sep 13, 2023
99f9af2
remove unused /SMB flag from ci.yml
martinzink Sep 13, 2023
eef6603
Revert "only warn in case of missing PutFile::ConflictResolution"
martinzink Sep 15, 2023
8c19ad1
increase time delta for MockSmbConnectionControllerService
martinzink Sep 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ endif()
# Enable usage of the VERSION specifier
if (WIN32)
add_compile_definitions(WIN32_LEAN_AND_MEAN _CRT_SECURE_NO_WARNINGS NOMINMAX)
add_compile_options(/W3 /utf-8)
add_compile_options(/W3 /utf-8 /bigobj)
endif()

if (NOT PORTABLE)
Expand Down
100 changes: 100 additions & 0 deletions PROCESSORS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ limitations under the License.
- [FetchOPCProcessor](#fetchopcprocessor)
- [FetchS3Object](#fetchs3object)
- [FetchSFTP](#fetchsftp)
- [FetchSmb](#fetchsmb)
- [FocusArchiveEntry](#focusarchiveentry)
- [GenerateFlowFile](#generateflowfile)
- [GetEnvironmentalSensors](#getenvironmentalsensors)
Expand All @@ -66,6 +67,7 @@ limitations under the License.
- [ListGCSBucket](#listgcsbucket)
- [ListS3](#lists3)
- [ListSFTP](#listsftp)
- [ListSmb](#listsmb)
- [LogAttribute](#logattribute)
- [ManipulateArchive](#manipulatearchive)
- [MergeContent](#mergecontent)
Expand All @@ -82,6 +84,7 @@ limitations under the License.
- [PutOPCProcessor](#putopcprocessor)
- [PutS3Object](#puts3object)
- [PutSFTP](#putsftp)
- [PutSmb](#putsmb)
- [PutSplunkHTTP](#putsplunkhttp)
- [PutSQL](#putsql)
- [PutTCP](#puttcp)
Expand Down Expand Up @@ -1027,6 +1030,36 @@ In the list below, the names of required properties appear in bold. Any other pr
| permission.denied | Any FlowFile that could not be fetched from the remote server due to insufficient permissions will be transferred to this Relationship. |


## FetchSmb

### Description

Fetches files from a SMB Share. Designed to be used in tandem with ListSmb.

### Properties

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

| Name | Default Value | Allowable Values | Description |
|---------------------------------------|---------------|------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **SMB Connection Controller Service** | | | Specifies the SMB connection controller service to use for connecting to the SMB server. |
| Input Directory | | | The full path of the file to be retrieved from the remote server. If left empty, the path and filename attributes of the incoming flow file will be used.<br/>**Supports Expression Language: true** |

### Relationships

| Name | Description |
|---------|--------------------------------------------------------------------|
| success | A flowfile will be routed here for each successfully fetched file. |
| failure | A flowfile will be routed here when failed to fetch its content. |

### Output Attributes

| Attribute | Relationship | Description |
|---------------|--------------|-------------------------------------------------------------------|
| error.code | failure | The error code returned by SMB when the fetch of a file fails. |
| error.message | failure | The error message returned by SMB when the fetch of a file fails. |


## FocusArchiveEntry

### Description
Expand Down Expand Up @@ -1689,6 +1722,48 @@ In the list below, the names of required properties appear in bold. Any other pr
| success | All FlowFiles that are received are routed to success |


## ListSmb

### Description

Retrieves a listing of files from an SMB share. For each file that is listed, creates a FlowFile that represents the file so that it can be fetched in conjunction with FetchSmb.

### Properties

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

| Name | Default Value | Allowable Values | Description |
|---------------------------------------|---------------|------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **SMB Connection Controller Service** | | | Specifies the SMB connection controller service to use for connecting to the SMB server. |
| Input Directory | | | The input directory to list the contents of |
| **Recurse Subdirectories** | true | | Indicates whether to list files from subdirectories of the directory |
| File Filter | | | Only files whose names match the given regular expression will be picked up |
| Path Filter | | | When Recurse Subdirectories is true, then only subdirectories whose path matches the given regular expression will be scanned |
| **Minimum File Age** | 5 sec | | The minimum age that a file must be in order to be pulled; any file younger than this amount of time (according to last modification date) will be ignored |
| Maximum File Age | | | The maximum age that a file must be in order to be pulled; any file older than this amount of time (according to last modification date) will be ignored |
| Minimum File Size | | | The minimum size that a file must be in order to be pulled |
| Maximum File Size | | | The maximum size that a file can be in order to be pulled |
| **Ignore Hidden Files** | true | | Indicates whether or not hidden files should be ignored |

### Relationships

| Name | Description |
|---------|-------------------------------------------------------|
| success | All FlowFiles that are received are routed to success |

### Output Attributes

| Attribute | Relationship | Description |
|------------------|--------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| filename | success | The name of the file that was read from filesystem. |
| path | success | The path is set to the relative path of the file's directory on the remote filesystem compared to the Share root directory. For example, for a given remote location smb://HOSTNAME:PORT/SHARE/DIRECTORY, and a file is being listed from smb://HOSTNAME:PORT/SHARE/DIRECTORY/sub/folder/file then the path attribute will be set to "sub/folder". |
| serviceLocation | success | The SMB URL of the share. |
| lastModifiedTime | success | The timestamp of when the file's content changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ss'. |
| creationTime | success | The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ss'. |
| lastAccessTime | success | The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ss'. |
| size | success | The size of the file in bytes. |


## LogAttribute

### Description
Expand Down Expand Up @@ -2360,6 +2435,31 @@ In the list below, the names of required properties appear in bold. Any other pr
| failure | FlowFiles that failed to send to the remote system; failure is usually looped back to this processor |


## PutSmb

### Description

Writes the contents of a FlowFile to an smb network location

### Properties

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

| Name | Default Value | Allowable Values | Description |
|---------------------------------------|---------------|-----------------------------|-----------------------------------------------------------------------------------------------------------------------|
| **SMB Connection Controller Service** | | | Specifies the SMB connection controller service to use for connecting to the SMB server. |
| Directory | . | | The output directory to which to put files<br/>**Supports Expression Language: true** |
| Conflict Resolution Strategy | fail | fail<br/>replace<br/>ignore | Indicates what should happen when a file with the same name already exists in the output directory |
| **Create Missing Directories** | true | | If true, then missing destination directories will be created. If false, flowfiles are penalized and sent to failure. |

### Relationships

| Name | Description |
|---------|-------------------------------------------------------------------------|
| success | All files are routed to success |
| failure | Failed files (conflict, write failure, etc.) are transferred to failure |


## PutSplunkHTTP

### Description
Expand Down
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,7 @@ Through JNI extensions you can run NiFi processors using NARs. The JNI extension
| ProcFs (Linux) | [ProcFsMonitor](PROCESSORS.md#procfsmonitor) | -DENABLE_PROCFS=ON |
| Python Scripting | [ExecuteScript](PROCESSORS.md#executescript)<br>[ExecutePythonProcessor](PROCESSORS.md#executepythonprocessor)<br/>**Custom Python Processors** | -DENABLE_PYTHON_SCRIPTING=ON |
| Sensors | GetEnvironmentalSensors<br/>GetMovementSensors | -DENABLE_SENSORS=ON |
| SMB (Windows) | [FetchSmb](PROCESSORS.md#fetchsmb)<br/>[ListSmb](PROCESSORS.md#listsmb)<br/>[PutSmb](PROCESSORS.md#putsmb) | -DENABLE_SMB=ON |
| SFTP | [FetchSFTP](PROCESSORS.md#fetchsftp)<br/>[ListSFTP](PROCESSORS.md#listsftp)<br/>[PutSFTP](PROCESSORS.md#putsftp) | -DENABLE_SFTP=ON |
| SQL | [ExecuteSQL](PROCESSORS.md#executesql)<br/>[PutSQL](PROCESSORS.md#putsql)<br/>[QueryDatabaseTable](PROCESSORS.md#querydatabasetable)<br/> | -DENABLE_SQL=ON |
| Splunk | [PutSplunkHTTP](PROCESSORS.md#putsplunkhttp)<br/>[QuerySplunkIndexingStatus](PROCESSORS.md#querysplunkindexingstatus) | -DENABLE_SPLUNK=ON |
Expand Down
1 change: 1 addition & 0 deletions cmake/MiNiFiOptions.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ if (WIN32)
add_minifi_option(MSI_REDISTRIBUTE_UCRT_NONASL "Redistribute Universal C Runtime DLLs with the MSI generated by CPack. The resulting MSI is not distributable under Apache 2.0." OFF)
add_minifi_option(ENABLE_WEL "Enables the suite of Windows Event Log extensions." OFF)
add_minifi_option(ENABLE_PDH "Enables PDH support." OFF)
add_minifi_option(ENABLE_SMB "Enables SMB support." ON)
lordgamez marked this conversation as resolved.
Show resolved Hide resolved
endif()

if(CMAKE_SYSTEM_NAME STREQUAL "Linux")
Expand Down
Loading