- IP headers parsing with IPv6 addresses:
Fix the parsing of IP headers such as X-Forwarded-For in presence of IPv6 addresses. Such addresses were mistakenly split by host and port number, later leading to IP parsing issues and thus ignoring the address.
-
Configuration:
Use the configuration keyproxy
both as an HTTP and HTTPS proxy setting. -
sdk/sqreen-instrumentation-tool
:
Fix filepath generation on Windows.
- Fix Sqreen's overhead rate calculation when the observed execution time is 0. This situation can happen when the execution timer wasn't precise enough in order to observe fast execution times.
- Add a new backend url configuration key
ingestion_url
for validation testing.
-
(#172) New SDK convenience function:
Add a new helper functionsdk.FromRequest()
allowing to retrieve Sqreen's request context and perform SDK calls. It is equivalent tosdk.FromContext(r.Context())
. -
(#156) Performance monitoring:
Monitor the execution time of requests protected by Sqreen. Optionally, it is possible to enforce the maximum amount of time Sqreen is allowed to run per request: Sqreen's monitoring and protections will only run for the given amount of time. This option is disabled by default and should be used with caution as it can lead to partially protected requests.
The resulting performance monitoring diagrams and setting are available at https://my.sqreen.com/application/goto/settings/performance.
Note that the execution time diagram cannot be used as a strict Application Performance Monitoring diagram as it is based on a lossy representation. It gives rough estimates of the actual execution time. -
(#170) Transparent response writer instrumentation:
Make the HTTP response writer instrumentation transparent by providing the same set of interfaces as the instrumented HTTP response writer. The set of interfaces is currently every optionalnet/http
response writer interface, along with some relevantio
interfaces, among which:http.Flusher
: for HTTP streaming support (multipart, chunked...).http.Pusher
: for HTTP2 server push support.http.Hijacker
: for websocket server support (experimental).io.ReaderFrom
: for optimized copies (eg. file copies)io.WriteString
: for optimized string copies.
-
(#163) HTTP status code 404 (not found) monitoring:
Automatically log a security event when the response status code is 404. This event is used by an internal Sqreen backend playbook to detect security scans. -
(#163) Scalable security event throughput:
To be able to handle a higher throughput of security events, the agent can now scale its number of goroutines. An extra goroutine is created every time the internal event queue is full, up to the number of available CPUs. Note that the agent still drops security events when the event queue is full in order to avoid slowing down the host application. -
(#165) Agent errors in the request hot-path:
To avoid slowing down request handlers, agent errors happening in the request hot path are now logged based on an exponential backoff algorithm.
This is disabled when the agent log level isdebug
.
- (#168) SDK return values:
The SDK function and method return values are no longer pointer values but Go interface values. This may break integrations using explicit SDK return types, and we recommend to instead use type-inference when possible. This change will allow us to transparently change the actual return values without involving any further breaking change.
As of today, the actual return value is a structure small enough to be returned by value in order to save memory-allocation and garbage-collection time. Returning an interface value allows to hide such implementation detail.
-
(#167) Playbook security response events:
Fix playbook security response events (blocking or redirecting a user or ip) so that Sqreen's dashboard can properly display them and link them to their source playbook. -
(#169) SQL-injection protection with Elastic APM:
Fix the detection of the SQL dialect when the SQL driver is instrumented by Elastic's APM tracer. This requires Elastic's Go agent version greater thanv1.9.0
. -
(#164) Echo middleware:
Fix the response status code monitoring when Echo's request handlers return an error. -
(#166) Gin middleware:
Fix the response content-length monitoring of default responses (ie. when the handler does nothing).
-
(#158) PII: make the PII scrubbing of In-App WAF attack events case-insensitive in order to correctly scrub transformed request parameters.
-
(#159) Monitoring: fix the content type and length monitoring of HTTP responses.
-
(#157) Gin middleware: use the request Go context instead of Gin's so that the agent can properly manage the request execution context, but also to correctly propagate values stored in the Go context before the middleware function.
- (#152, #155) Add In-App WAF protection to Echo's request parameter parser:
Context
's methodBind()
is now protected by the In-App WAF. The Go value it parses from the HTTP request is made available to the In-App WAF rules via theGET/POST parameters
field.
When blocked,Bind()
returns a non-nilSqreenError
value and its caller should immediately return.
Read more about the blocking behavior of Sqreen for Go at https://docs.sqreen.com/go/integration.
- (#153) RASP shellshock: properly handle environment variables containing
variable definitions (eg.
TERMCAP
).
- (#149) Add activity monitoring of RASP protections. The number of times a
protection has been used in the application is now displayed on the
Activity
tab of each RASP protection dashboard page.
-
(#148) Fix the usage of the Go agent in a reverse proxy server: avoid automatically reading a POST request's body because of the former usage of
Request.ParseForm()
in Sqreen's middleware functions, and rather get POST form values fromRequest.PostForm
, and URL query values fromRequest.URL.Query()
.
Note that sinceRequest.PostForm
's value is assigned byRequest.ParseForm()
, the In-WAF and RASP protections will now only consider POST form values when the request handler will have calledRequest.ParseForm()
itself for its own needs.
Therefore, the In-App WAF is now also attached toParseForm()
to monitor the resulting POST form values, and returns a non-nil error when an attack is detected (cf. https://docs.sqreen.com/go/integration for more Go integration details) . -
(ef81fc2) Enforce a request body reader to ignore it when blocked by the In-App WAF by returning it 0 bytes read along with the current non-nil error. This allows for example
io.LimitReader
not to copy the body buffer despite the non-nil error returned by the In-App WAF protection.
- (#142) RASP: add Shellshock protection support. This protection is currently
attached to
os.StartProcess()
which is the common function of the Go standard library to execute a process. This protection can be configured at https://my.sqreen.com/application/goto/modules/rasp/details/shellshock.
-
(#145) In-App WAF: always recover from panics as this in the way the
reflect
package handles usage errors. -
(#144) Backend client: avoid dropping HTTP traces in case of
Host
header parsing errors.
- (#137) RASP: add noSQL Injection protection support for the Go MongoDB driver
go.mongodb.org/mongo-driver/mongo
. This protection can be configured at https://my.sqreen.com/application/goto/modules/rasp/details/nosql_injection.
-
(#138) Health-check the HTTPS connectivity to the new backend API
ingestion.sqreen.com
before using it. Fallback to the usualback.sqreen.com
in case of a connection issue. Therefore, the agent can take up to 30 seconds to connect to Sqreen if the health-check timeouts. Please make sure to add this new firewall and proxy configurations. -
(#136) Add support to attach multiple security protections per hook point.
- (#140) Fix the In-App WAF metadata PII scrubbing to also match substrings.
-
(d81222d) Add missing request parameters when both JSON values and form values were present - only the form values were taken into account.
-
(ee22b77) Upgrade to libsqreen v0.7.0:
- Fix false positives in libinjection SQL heuristics.
- Fix a false positive in libinjection XSS heuristics.
- Add support for boolean values.
- Add support for float values.
- Fix memory deallocator of scalar values.
-
(c425760) Fix data bindings with null values.
- (eeb1dca) Avoid copying the metadata returned by the In-App WAF.
-
(#130) In-App WAF protection of the HTTP request body:
Since the HTTP request handler needs to explicitly read the request body, and ultimately parse it into a Go value, the In-App WAF is now applied to new points in the request lifecycle:-
Reading the request body is now monitored until EOF is reached, and the raw body bytes are made available to the In-App WAF rules via a new In-App WAF field
Body
. Note that such In-App WAF rules can be created on custom In-App WAF rulesets only. -
Parsers can be now protected by the In-App WAF once they have parsed a request input into a Go value. The parsed value is made available to the In-App WAF rules via the
GET/POST parameters
field. Every existing In-App WAF rule using this field therefore applies.
This new feature is firstly deployed on Gin'sShouldBind()
method which is Gin's function to parse HTTP request values. It allows to cover every parser Gin provides such asBindJSON()
,BindXML()
, etc.When blocked, the function returns a non-nil
SqreenError
value and the caller should immediately return.
Read more about the blocking behavior of Sqreen for Go at https://docs.sqreen.com/go/integration.
-
-
(#129) Update Sqreen's blocking HTML page with a clearer description of what users getting it should do.
- (794d6e2) Allow port numbers in the
X-Forwarded-For
header.
-
(#119) RASP: add Shell Injection protection support. This protection is currently dynamically applied to
os.StartProcess()
which is the only entry point of the Go standard library to execute a process. This protection can be configured at https://my.sqreen.com/application/goto/modules/rasp/details/shi. -
(#119) RASP: add Local File Inclusion protection support. This protection is currently dynamically applied to
os.Open()
which is the only entry point of the Go standard library to open a file for reading. This protection can be configured at https://my.sqreen.com/application/goto/modules/rasp/details/lfi. -
(#120) RASP: add Server-Side Request Forgery protection support. This protection is currently dynamically applied to
net/http.(*Client).do()
which is the only entry point of the Go standard library to perform an HTTP request. This protection can be configured at https://my.sqreen.com/application/goto/modules/rasp/details/ssrf. -
(#125) RASP: enable SQL Injection protection for every MySQL, Oracle, SQLite and PostgreSQL drivers listed in the Go language wiki page https://github.com/golang/go/wiki/SQLDrivers.
-
(#115) RASP: store Sqreen's request protection context into the Goroutine Local Storage (GLS). Therefore, Sqreen can now protect every Go function without requiring the request Go context (eg. both
QueryContext()
andQuery()
can be now protected against SQL injections). For now, this protection context is only available in the goroutine handling the request, and sub-goroutines are not protected. Further support will be added very soon to remove this limitation. -
(#121) Add IP denylist support: block every request performed by an IP address of the denylist. Every usage of whitelist and blacklist in the agent was also removed when possible. The IP denylist can be configured at https://my.sqreen.com/application/goto/settings/denylist.
-
(#122) Add path passlist support: requests performed on those paths are not monitored nor protected by Sqreen. The Path passlist can be configured at https://my.sqreen.com/application/goto/settings/passlist.
-
(#123) Export the error type returned by Sqreen protections when blocking in the new SDK package
github.com/sqreen/go-agent/sdk/types
in order to avoid retrying blocked function calls (eg. avoid retrying a blocked SQL query). It must be used along witherrors.As()
to detect such cases. Read more at https://godoc.org/github.com/sqreen/go-agent/sdk/types. -
(#124) Allow to "quickly" remove the agent from a program by only removing it from the source code without disabling the program instrumentation. This is made possible by making the instrumentation fully autonomous to avoid compilation errors.
- Gin Middleware: fix the HTTP status code monitoring that was possibly changed by Gin after having been already written.
-
(#126) Cache request value lookups, mainly to accelerate the In-App WAF when lots of rulesets are enabled.
-
(#117) Simpler Go vendoring support implementation.
-
(#113) Significant JavaScript performance improvements by changing the virtual machine to
github.com/dop251/goja
. -
(#114) Add Goroutine Local Storage (GLS) support through static instrumentation of the Go runtime.
- (#116) Fix the instrumentation tool ignoring vendored packages, leading to missing hook points in the agent.
-
(#109) Make the PII sanitizer configurable with two new configuration entries allowing to control the regular expressions used to sanitize everything sent to Sqreen. The agent doesn't start in case of an invalid regular expression. More details can be found on the configuration's documentation page .
-
(#110) The
net/http
middleware now includes URL segments in the request parameters to increase the coverage we have on frameworks compatible with it, such asgorilla
orbeego
.
- (#107) Backend API: integrate the security signal HTTP API.
-
(#108) Update the token validation to correctly handle the new token format.
-
(#111) Fix the JSON serialization function of HTTP headers monitored by the agent that could fail depending on the header values. Note that the JSON serialization of the parent data structure safely catches any JSON injection attempt.
-
Add quick start examples for common build and deployment environments such as docker images, heroku and google app engine.
-
Document Heroku installation at https://docs.sqreen.com/go/installation/heroku.
-
Document Google App Engine installation at https://docs.sqreen.com/go/installation/google-app-engine.
-
Document Docker image installation at https://docs.sqreen.com/go/installation/docker.
-
Document PII scrubbing configuration at https://docs.sqreen.com/go/configuration/#personally-identifiable-information-scrubbing .
-
(#99) Fix mistakenly enforced HTTP status code
200
when Sqreen's middleware function is not the first in the request handling chain. This issue appeared when not adding Sqreen's middleware function as the root HTTP middleware. -
(#100) Fix the monitoring of HTTP response codes mistakenly considered
200
when set by the request handlers. -
(#101) Prevent starting the agent when the instrumentation tool and agent versions are not the same.
This new major version says farewell to the beta
and adds SQL-injection run
time protection thanks the first building blocks of RASP
for the Go language! Thank you to everyone that helped us in this wonderful and
amazing journey ;-)
The Go agent has been protecting production servers for more than a year now and we reached the point where we are confident enough about its deployment, setup, but also its internals and specific integrations with the Go language and runtime.
We are getting closer to the fully-featured agent v1.0 as we will now be able to fully add support for every RASP protection Sqreen supports.
Because we now want a stable public API, find below the breaking changes:
-
The former separate agent package
github.com/sqreen/go-agent/agent
that was asked to import in order to start the agent is no longer required nor available. This is now performed by the middleware functions we provide in order to avoid the most common setup mistake during the beta where only the agent was setup and no middleware function was set to protect the requests ( and therefore nothing was happening). -
SDK: the user identification SDK method
Identify()
has been updated to be simpler to use and less error-prone by now making it return a non-nil error when the request handler shouldn't continue any further serving the request. It happens when a user security response has matched the identified user. This replaces the former separate SDK methodMatchSecurityResponse()
. New usage example:sqUser := sq.ForUser(sdk.EventUserIdentifiersMap{"uid": "unique user id"}) if err := sqUser.Identify(); err != nil { return }
-
The agent no longer starts if the program wasn't instrumented using the instrumentation tool. See docs.sqreen.com/go/installation for details on how to install and use the tool. Note that the program is not aborted - only the agent is disabled.
-
Dropping gRPC support: the beta support for gRPC was experimental and was in the end too limited by Sqreen's focus on the HTTP protocol. Most of our protections are indeed designed for HTTP and couldn't be applied at the gRPC protocol level. We are therefore removing it until we can provide a correct experience for such HTTP-based protocol.
Please contact us if you need any further information or if you are interested in helping us building it ([email protected]).
-
SQL-injection RASP protection: when enabled on Sqreen's dashboard, the
database/sql
Go package gets automatically protected against SQL injections. SQL queries go through our SQL-injection detection which will abort the SQL function call and corresponding HTTP request when an attack is detected.
Note that special care was taken to properly intergrate with Go error-handling principles: when a SQL query gets blocked, the HTTP request context is canceled and a non-nil error is returned by thedatabase/sql
function call in order to fall into the existing error-handling flow. For example:// The following query can be injected. An error is returned when the SQL // query was blocked. rows, err := db.QueryContext(ctx, "select id, name from users where id=" + unsafe) if err != nil { return err }
Read more about Go integration details at http://docs.sqreen.com/go/integration.
-
Dashboard diagnostic messages: major setup issues are now also reported through Sqreen's dashboard page of running hosts to get notified about some downgraded states of the agent, such as:
- The Go program is not instrumented so the agent didn't start.
- The In-App WAF wasn't compiled (eg. CGO disabled) so it is unavailable and disabled.
- The program dependencies couldn't be retrieved because the program was not compiled as a Go module. This is also shown by the dashboard when the list of dependencies is empty.
-
(#89) Go instrumentation: Sqreen's dynamic configuration of the protection your Go programs is made possible at run time thanks to Go instrumentation. It is a building block of the upcoming run time self-protection (aka RASP) and it is safely performed at compilation time by an instrumentation tool that seamlessly integrates with the Go toolchain. To begin with, only a specific set of Go packages are instrumented: the agent and
database/sql
(to prepare the upcoming SQL injection protection).Please, find out how to install and use the tool on the new agent installation documentation available at https://docs.sqreen.com/go/installation/.
-
(#90) The SDK now imports the agent package to no longer have to import it in the
main
package. The SDK is indeed mandatory when setting up Sqreen for Go, making it the best place to import the agent. -
(#91) The program dependencies are now sent to Sqreen to perform dependency analysis (outdated, vulnerable, etc.). They are only available when the Go program you compile is a Go module. Sqreen's dashboard Dependency page will made available be soon.
- (#92) Vendoring using
go mod vendor
could lead to compilation errors due to missing files.
-
Request parameters such as query or post parameters are now added in the attack events and shown in the attack logs and in the event explorer pages of our dashboard. (#84)
-
PII scrubbing is now performed on every event sent to Sqreen, as documented on https://docs.sqreen.com/guides/how-sqreen-works/#pii-scrubbing. (#86)
- Add PII scrubbing to the WAF logs that may include data from the request. (
- The In-App WAF has been intensively optimized so that large requests can no longer impact its execution time. (#83)
- In-App WAF:
- Dynamically set the WAF timeout (#79).
- Ignore WAF timeout errors and add more context when reporting an error (
- Update the libsqreen to v0.4.0 to add support for the
@pm
operator.
- CGO bindings are now involved in the compilation of the agent and will require
the
gcc
compiler and the C library headers. Note that compiling the agent without CGO (CGO_ENABLED=0
) is still possible but will disable some agent features; in this case the new WAF feature described below.
- Add support for the In-App WAF: an out-of-the-box Web-Application Firewall leveraging the full application context, that is fail-safe, has limited false positives and won’t require heavy fine-tuning. Only darwin/amd64 and linux/amd64 targets are supported so far. Any other target will get this feature disabled. More targets will be added in future versions. (#77)
- Increase the internal timeout value of the HTTP client to Sqreen's backend in order to be more resilient to normal networking delays.
- Fix a compilation error on 32-bit target architectures.
-
Fully-featured playbooks with the added ability into the agent to redirect the request to a given URL. (#72)
-
Configurable protection behaviour of the agent when blocking a request by either customizing the HTTP status code that is used for the blocking HTML page, or by redirecting to a given URL instead.
Dashboard page: https://my.sqreen.com/application/goto/settings/global#protection-mode -
HTTP response status code monitoring. (#75)
Dashboard page: https://my.sqreen.com/application/goto/monitoring -
Support for browser security headers protection modules allowing to enable various browser security options allowing to restrict modern browsers from running into some preventable vulnerabilities:
-
Content Security Policy protection module allowing to prevent cross-site scripting attacks. (#74)
Dashboard page: https://my.sqreen.com/application/goto/modules/csp -
Security headers protection module allowing to protect against client-side vulnerabilities in the browser. (#73)
Dashboard page: https://my.sqreen.com/application/goto/modules/headers
-
- Better agent configuration logs clearly stating where does the configuration
come from (file in search path, enforced file or environment variables), along
with the possibility to display the full settings using the
debug
log-level.
-
Middleware functions, called interceptors, for gRPC over HTTP2. More details on how to use it at https://godoc.org/github.com/sqreen/go-agent/sdk/middleware/sqgrpc. (#23)
-
IP whitelist support to make the agent completely ignore requests whose IP addresses are whitelisted. Everything related to Sqreen, including events, will be ignored. (#69)
-
Agent fail-safe catching errors and panics in order to prevent the host Go app to fail. The fail-safe mechanism either tries to restart the agent or ultimately stops it. (#67)
- Internal event batch improvements:
- Increased batch buffer capacity from 60 to 6000 entries in order to be able to handle more events, sent by batches of 60 events per heartbeat.
- Remove a bookkeeping goroutine and include its logic into the main event processing loop.
This release adds the ability to block IP addresses or users into your Go web services by adding support for Security Automation according to your playbooks and their configured security responses.
Note that redirecting users or IP addresses is not supported yet.
-
Security Automation:
It is now possible to block IP addresses or users. When a playbook triggers, the agent is notified and gets the batch of security responses. They are asynchronously stored into data structures optimized for fast lookup and low memory usage. Middleware functions can thus perform fast lookups to block requests in a few microseconds in order to exit request handlers as fast as possible.-
Blocking IP addresses:
No changes are required to block IP addresses. Our middleware functions have been updated to block requests whose IP addresses match a security response. The request is aborted with HTTP status code500
and Sqreen's default HTML information page. -
Blocking users:
Blocking users is performed by combining SDK methodsIdentify()
andMatchSecurityResponse()
in order to firstly associate a user to the current request, and secondly to check if it matches a security response. When a security response matches, the request handler and any related goroutines should be stopped as soon as possible.Usage example:
uid := sdk.EventUserIdentifiersMap{"uid": "my-uid"} sqUser := sdk.FromContext(ctx).ForUser(uid) sqUser.Identify() if match, err := sqUser.MatchSecurityResponse(); match { // Return now to stop further handling the request and let Sqreen's // middleware apply the configured security response and abort the // request. The returned error may help aborting from sub-functions by // returning it to the callers when the Go error handling pattern is // used. return err }
We strongly recommend to create a user-authentication middleware function in order to seamlessly integrate user-blocking to all your user-authenticated endpoints.
-
- Escape the event type name to avoid JSON marshaling error. Note that this case could not happen in previous agent versions. (#52)
- Avoid performing multiple times commands within the same command batch. (51)
-
Get the client IP address out of the HAProxy header
X-Unique-Id
using the new configuration variableip_header_format
. (#41) -
New configuration option
strip_http_referer
/SQREEN_STRIP_HTTP_REFERER
allowing to avoid sending theReferer
HTTP header to the Sqreen backend when it contains sensitive data. (#36) -
Ability to disable/enable the agent through the dashboard settings using the Sqreen status button. (#29)
- Agent internals are now under a private Go package and can no longer be
imported. Any sub-package under
github.com/sqreen/go-agent/agent
was not supposed to be imported and is now private to avoid future confusions. (#27)
-
Remove duplicate
User-Agent
entry sent twice in the request record. (#42) -
Fix IPv4 and IPv6 matching against private network definitions. (#38)
-
Remove useless empty request records mistakenly created while not carrying any SDK observation. (#38)
- Better memory management and footprint when the agent is disabled by removing globals. This will be also required to be able to cleanly restart the agent by self-managing the initializations. (#28)
- Add a new
Identify()
method allowing to explicitly associate a user to the current request. As soon as we add the support for the security reponses, it will allow to block users (#26).
This version is a new major version towards the v0.1.0 as it proposes a new and stable SDK API, that now will only be updated upon user feedback. So please, share your impressions with us.
-
New web framework middleware support:
- Standard Go's
net/http
package (#21). - Echo (#19).
- Standard Go's
-
Multiple custom events can now be easily associated to a user using the user-scoped methods under
ForUser()
. For example, to send two custom events for a given user, do:sqUser := sqreen.ForUser(uid) sqUser.TrackEvent("my.event.one") sqUser.TrackEvent("my.event.two")
-
The configuration file can now be stored into multiple locations, the current working directory or the executable one, or enforced using the new configuration environment variable
SQREEN_CONFIG_FILE
(#25). -
The custom client IP header configured in
SCREEN_IP_HEADER
is now also sent to Sqreen so that it can better understand what IP headers were considered by the agent to determine what is the actual client IP address (67e2d4cbf9b883e9e91e1a5d9e53348a18c1b900).
-
Stable SDK API of "Sqreen for Go":
-
Avoid name conflicts with framework packages by prefixing Sqreen's middleware packages with
sq
. For example,gin
becomessqgin
(#17). -
Cleaner Go documentation now entirely included in the SDK and middleware packages Go documentations. So no more need to go inside the agent documentation to know more on some SDK methods, it is now all documented in the same place, with lot of examples.
-
Clearer SDK API: The flow of security events that can send to Sqreen is now well-defined by a tree of SDK methods that can only be used the right way. (#18, #24)
-
The SDK handle getter function name is renamed from
GetHTTPRequestContext()
into a simplerFromContext()
. -
User-related SDK methods are now provided by
ForUser()
, for example:sqreen.TrackAuth(true, uid)
becomes
sqreen.ForUser(uid).TrackAuthSuccess()
-
-
-
sdk: user-related security events:
- ability to associate a user to an event using
WithUserIdentifier()
(#13) . - track user creation using
TrackSignup()
(#15). - track user authentication usingTrackAuth()
(#15).
- ability to associate a user to an event using
-
agent/backend: take into account
{HTTPS,HTTP,NO}_PROXY
environment variables (and their lowercase alternatives) (#14). -
agent/backend: share the organization token for all your apps (#12).
- agent/config: avoid conflicts with global viper configs (#16).
- sdk: better documentation with examples.