Skip to content

[Filebeat][okta] Fix okta pagination #21797

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 16, 2020
Merged

[Filebeat][okta] Fix okta pagination #21797

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3bccb29
Fix okta pagination
marc-gr Oct 14, 2020
50c6cef
Use cursor storage
marc-gr Oct 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion x-pack/filebeat/input/httpjson/date_cursor.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,9 @@ func newDateCursorFromConfig(config config, log *logp.Logger) *dateCursor {
c.urlField = config.DateCursor.URLField
c.initialInterval = config.DateCursor.InitialInterval
c.dateFormat = config.DateCursor.getDateFormat()
c.valueTpl = config.DateCursor.ValueTemplate.Template
if config.DateCursor.ValueTemplate != nil {
c.valueTpl = config.DateCursor.ValueTemplate.Template
}

return c
}
Expand Down
2 changes: 1 addition & 1 deletion x-pack/filebeat/input/httpjson/pagination.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ func (p *pagination) nextRequestInfo(ri *requestInfo, response response, lastObj

// getNextLinkFromHeader retrieves the next URL for pagination from the HTTP Header of the response
func getNextLinkFromHeader(header http.Header, fieldName string, re *regexp.Regexp) (string, error) {
links, ok := header[fieldName]
links, ok := header[http.CanonicalHeaderKey(fieldName)]
Copy link
Member

@andrewkroh andrewkroh Oct 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if the service didn't implement its headers using the canonical form? Should our code look for both the value that the user configured and the canonical form (if they differ)?

Copy link
Contributor Author

@marc-gr marc-gr Oct 15, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think from the docs http.Header keys are always in the canonical form (https://golang.org/pkg/net/http/#Header, https://golang.org/src/net/http/response.go), I added this to take this into account since we are accessing the map directly instead of using the Get method which takes care of doing it itself.

if !ok {
return "", fmt.Errorf("field %s does not exist in the HTTP Header", fieldName)
}
Expand Down
2 changes: 1 addition & 1 deletion x-pack/filebeat/input/httpjson/pagination_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import (

func TestGetNextLinkFromHeader(t *testing.T) {
header := make(http.Header)
header.Add("Link", "<https://dev-168980.okta.com/api/v1/logs>; rel=\"self\"")
header.Add("link", "<https://dev-168980.okta.com/api/v1/logs>; rel=\"self\"")
header.Add("Link", "<https://dev-168980.okta.com/api/v1/logs?after=1581658181086_1>; rel=\"next\"")
re, _ := regexp.Compile("<([^>]+)>; *rel=\"next\"(?:,|$)")
url, err := getNextLinkFromHeader(header, "Link", re)
Expand Down
7 changes: 4 additions & 3 deletions x-pack/filebeat/input/httpjson/requester.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,7 @@ func (r *requester) processHTTPRequest(ctx context.Context, publisher cursor.Pub
return err
}

response.header = resp.Header
responseData, err := ioutil.ReadAll(resp.Body)
if err != nil {
return fmt.Errorf("failed to read http response: %w", err)
Expand Down Expand Up @@ -165,10 +166,10 @@ func (r *requester) processHTTPRequest(ctx context.Context, publisher cursor.Pub
if err != nil {
return err
}
}

if lastObj != nil && r.dateCursor.enabled {
r.updateCursorState(ri.url, r.dateCursor.getNextValue(common.MapStr(lastObj)))
if lastObj != nil && r.dateCursor.enabled {
r.updateCursorState(ri.url, r.dateCursor.getNextValue(common.MapStr(lastObj)))
}
}

return nil
Expand Down
4 changes: 4 additions & 0 deletions x-pack/filebeat/module/okta/system/config/input.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,10 @@ ssl: {{ .ssl | tojson }}
url: {{ .url }}
{{ end }}

date_cursor.field: published
date_cursor.url_field: since
date_cursor.initial_interval: {{ .initial_interval }}

{{ else if eq .input "file" }}

type: log
Expand Down
3 changes: 3 additions & 0 deletions x-pack/filebeat/module/okta/system/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ var:
default: "SSWS"
- name: http_client_timeout
- name: http_method
default: GET
- name: http_headers
- name: http_request_body
- name: interval
Expand All @@ -31,6 +32,8 @@ var:
- name: tags
default: [forwarded]
- name: url
- name: initial_interval
default: 24h

input: config/input.yml
ingest_pipeline: ingest/pipeline.yml
Expand Down