Tomcat security and API

[unhipzippo]

(Testing on OpenGrok 1.2.24, AdoptOpenJDK 11.0.3, Tomcat 9.0.21 on Oracle Linux 7.6)

If OpenGrok is running inside a Tomcat instance which uses Tomcat security to protect the entire OpenGrok webapp (rather than using the OpenGrok-specific authorization framework), the Tomcat security defaults to extending to the OpenGrok API as well. Since the OpenGrok indexer's call to the configuration API endpoint doesn't expect/include authorization, this results in indexer completion never informing the webapp of the new configuration.

Somewhat related issues:

#2352
#2635

In order to permit the indexer to update the webapp with new configuration, https://github.com/oracle/opengrok/wiki/Authorization recommends exempting API calls from security entirely:

   <security-constraint>
       <web-resource-collection>                                               
           <web-resource-name>API endpoints are checked separately by the web app</web-resource-name>
           <url-pattern>/api/*</url-pattern>                                   
       </web-resource-collection>                                              
   </security-constraint>

However, since the API search endpoint can return data which may be considered privileged (e.g. lines of source code), permitting completely public access to the search / suggest endpoints might not be ideal. (@tulinkry alludes to this in #2635 (comment))

Is there a recommended alternative configuration if we want to apply security to API calls?

[unhipzippo]

One possibility we're experimenting with is some tweaks to both Tomcat and to our preexisting front-end proxy (nginx):

• Proxy (nginx) on port :80, OpenGrok listening on localhost only, e.g. localhost:8080.
  e.g. Tomcat server.xml:

    <Connector address="127.0.0.1" port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               maxHttpHeaderSize="16384"
               redirectPort="8443" />

• Add front-end proxy rules to explicitly block all paths under /<webapp_context/api/v1/ EXCEPT for search, suggest and suggest/config endpoints.
  e.g. nginx .../conf/vhosts/opengrok.conf:

upstream opengrok-main-http-backend {
    server 127.0.0.1:8080;
    keepalive 10;
    keepalive_timeout 15s;
}

server {
    server_name opengrok.company.com;

    listen *:80;

    location /source/ {

        location ~ ^/source/api/v1/(?!(search|suggest(/config)?)$) {
            allow 127.0.0.1;
            deny all;

            proxy_pass http://opengrok-main-http-backend;
        }

        proxy_pass http://opengrok-main-http-backend;
    }
}

• With this proxy change in place, tweak Tomcat security constraint on /source/ webapp to ONLY exempt the API config endpoint from auth enforcement, not all API calls:

    <security-constraint>
        <!--
             Explicitly permit /api/v1/configuration without applying our
             below password restriction on it, since the localhost indexer
             needs to call the API without authentication to pass post-index
             config to the webapp.
        -->
        <web-resource-collection>
            <web-resource-name>Auth-exempt area (API config)</web-resource-name>
            <url-pattern>/api/v1/configuration</url-pattern>
        </web-resource-collection>
    </security-constraint>
    <security-constraint>
        <web-resource-collection>
            <web-resource-name>Auth-required area</web-resource-name>
            <url-pattern>/*</url-pattern>
        </web-resource-collection>
        <auth-constraint>
            <role-name>${ldap.group}</role-name>
        </auth-constraint>
    </security-constraint>
    <login-config>
        <auth-method>BASIC</auth-method>
        <realm-name>Username and Password</realm-name>
    </login-config>
    <security-role>
        <role-name>${ldap.group}</role-name>
    </security-role>

This should permit the indexer to issue config endpoint calls without requiring authentication (from localhost only), but all other API calls (external or from localhost) should be forced to authenticate via LDAP (and be members of ${ldap.group}, passed in at Tomcat runtime via -Dldap.group)

I haven't yet migrated our instance config to the OpenGrok-specific authorization framework introduced post 1.0, so I don't know offhand if it applies authorization to API calls, or if it similarly permits them without requiring authorization.

[tulinkry]

This should permit the indexer to issue config endpoint calls without requiring authentication (from localhost only), but all other API calls (external or from localhost) should be forced to authenticate via LDAP (and be members of ${ldap.group}, passed in at Tomcat runtime via -Dldap.group)

This way you can use only full index run, not partial index nor repository synchronization scripts. But that is not your use case.

I haven't yet migrated our instance config to the OpenGrok-specific authorization framework introduced post 1.0, so I don't know offhand if it applies authorization to API calls, or if it similarly permits them without requiring authorization.

OpenGrok specific authorization wouldn't solve this for you. It aims to solve per project or per group authorization. The json search request is under this specific authorization when it's in place.

[vladak]

However, since the API search endpoint can return data which may be considered privileged (e.g. lines of source code), permitting completely public access to the search / suggest endpoints might not be ideal. (@tulinkry alludes to this in #2635 (comment))

TLDR: It cannot. The /search endpoint is properly authorized.

Long answer: The built in authorization is a bit of a complex beast. There are some related filters defined in the web.xml file (like AuthorizationFilter) however the /search API endpoint is authorized differently. SearchController calls engine.search() which ends up in

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/search/SearchEngine.java

Lines 293 to 307 in 39bbbcb

	public int search(HttpServletRequest req, String... projectNames) {
	ProjectHelper pHelper = PageConfig.get(req).getProjectHelper();
	Set<Project> allProjects = pHelper.getAllProjects();
	List<Project> filteredProjects = new ArrayList<Project>();
	for(Project project: allProjects) {
	for (String name : projectNames) {
	if (project.getName().equalsIgnoreCase(name)) {
	filteredProjects.add(project);
	}
	}
	}
	return search(
	filteredProjects,
	new File(RuntimeEnvironment.getInstance().getDataRootFile(), IndexDatabase.INDEX_DIR));
	}

and when the above code calls pHelper.getAllProjects() it will eventually bubble down to

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/web/ProjectHelper.java

Lines 217 to 224 in 39bbbcb

	private Set<Project> cacheProjects(String name, Set<Project> original) {
	Set<Project> p = (Set<Project>) cfg.getRequestAttribute(name);
	if (p == null) {
	p = filterProjects(original);
	cfg.setRequestAttribute(name, p);
	}
	return p;
	}

(or cacheGroups()) and the actual authorization happens inside filterProjects() (or filterGroups()) called from there.

[tulinkry]

But that is the authorization framework, that is correctly working. But the question was different - the user has LDAP setup (in tomcat) to authenticate/authorize users to access the opengrok instance. Which isn't working for /api/v1/search because this endpoint is excluded from this LDAP check as adviced on our wiki.

[vladak]

I see. In general I'd deter anyone from doing authorization elsewhere than inside OpenGrok. There are just too many surface points to cover to do this correctly, plus it would be reinventing the wheel. So, from the point of view of the web app, authentication needs to be done externally, authorization internally.

[tulinkry]

I also think that the author here mixes authentication with authorization, so what is this about is mainly authentication. It is still valid that unauthenticated user can access /api/v1/search and thus expose the forbidden content.

[vladak]

Well, authentication is performed first so that authorization can follow. If authorization was done using the built in framework, it would not permit unauthenticated requests to pass through.

[vladak]

That said, the documentation seems to be wrong. I wonder if it is possible to instruct Tomcat to perform authentication only and pass authentication headers to the web application that would then perform authorization. https://stackoverflow.com/questions/7756048/authentication-without-authorization-on-tomcat-7 gives some hope.

[vladak]

Adding this to webapps/source/WEB-INF/web.xml:

    <security-constraint>
        <web-resource-collection>
            <web-resource-name>In general everything needs to be authenticated</web-resource-name>
            <url-pattern>/*</url-pattern> <!-- protect the whole application -->
        </web-resource-collection>

        <auth-constraint>
            <role-name>*</role-name>
        </auth-constraint>

        <user-data-constraint>
            <transport-guarantee>NONE</transport-guarantee>
        </user-data-constraint>
    </security-constraint>

    <security-role>
        <role-name>*</role-name>
    </security-role>

    <login-config>
        <auth-method>BASIC</auth-method>
    </login-config>

plus having conf/tomcat-users.xml to have these contens:

<?xml version='1.0' encoding='utf-8'?>
<tomcat-users xmlns="http://tomcat.apache.org/xml"
              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              xsi:schemaLocation="http://tomcat.apache.org/xml tomcat-users.xsd"
              version="1.0">

  <user username="foo" password="bar" roles="tomcat,manager-script"/>
</tomcat-users>

triggers authentication but not authorization in Tomcat.

[vladak]

I think we need to introduce proper HTTP Basic Auth decoder for this to be useful. The supplied HttpBasicAuthorizationPlugin is for demonstration purposes only.