spring.ftl doesn't work on freemarker 2.3.24+'s auto escaping feature [SPR-14740]

[spring-projects-issues]

Tokuhiro Matsuno opened SPR-14740 and commented

spring boot's spring.ftl is using ?html (legacy escaping) in some macros.

As a result, if user enables auto escaping feature by -Dspring.freemarker.settings.output_format=HTMLOutputFormat, <@spring.formInput "filter.identityPrincipal", 'size="20" maxlength="20"'/> throws exception.

FreeMarker template error (HTML_DEBUG mode; use RETHROW in production!)

Lazy initialization of the imported namespace for "spring.ftl" has failed; see cause exception

----
FTL stack trace ("~" means nesting-related):
    - Failed at: @spring.formInput "filter.identityPri...  [in template "spring_ftl.ftl" at line 4, column 5]
    ~ Reached through: #nested  [in template "__wrapper.ftl" in macro "main" at line 15, column 5]
    ~ Reached through: @wrapper.main  [in template "spring_ftl.ftl" at line 2, column 1]
----

Java stack trace (for programmers):
----
freemarker.template.TemplateModelException: [... Exception message was already printed; see it above ...]
    at freemarker.core.Environment$LazilyInitializedNamespace.ensureInitializedTME(Environment.java:2882)
    at freemarker.core.Environment$LazilyInitializedNamespace.get(Environment.java:2939)
    at freemarker.core.Dot._eval(Dot.java:43)
    at freemarker.core.Expression.eval(Expression.java:81)
    at freemarker.core.UnifiedCall.accept(UnifiedCall.java:73)
    at freemarker.core.Environment.visit(Environment.java:363)
    at freemarker.core.Environment.invokeNestedContent(Environment.java:572)
    at freemarker.core.BodyInstruction.accept(BodyInstruction.java:60)
    at freemarker.core.Environment.visit(Environment.java:363)
    at freemarker.core.Environment.invoke(Environment.java:715)
    at freemarker.core.UnifiedCall.accept(UnifiedCall.java:83)
    at freemarker.core.Environment.visit(Environment.java:327)
    at freemarker.core.Environment.visit(Environment.java:333)
    at freemarker.core.Environment.process(Environment.java:306)
    at freemarker.template.Template.process(Template.java:386)
    at org.springframework.web.servlet.view.freemarker.FreeMarkerView.processTemplate(FreeMarkerView.java:367)
    at org.springframework.web.servlet.view.freemarker.FreeMarkerView.doRender(FreeMarkerView.java:284)
    at org.springframework.web.servlet.view.freemarker.FreeMarkerView.renderMergedTemplateModel(FreeMarkerView.java:234)
    at org.springframework.web.servlet.view.AbstractTemplateView.renderMergedOutputModel(AbstractTemplateView.java:167)
    at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:303)
    at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1257)
    at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1037)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:980)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:87)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:522)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
    at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:677)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:1110)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:785)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1425)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
Caused by: freemarker.core.ParseException: Syntax error in template "spring.ftl" in line 225, column 36:
Using ?html (legacy escaping) is not allowed when auto-escaping is on with a markup output format (HTML), to avoid double-escaping mistakes.
    at freemarker.core.FMParser.BuiltIn(FMParser.java:1193)
    at freemarker.core.FMParser.AddSubExpression(FMParser.java:1101)
    at freemarker.core.FMParser.PrimaryExpression(FMParser.java:593)
    at freemarker.core.FMParser.UnaryExpression(FMParser.java:639)
    at freemarker.core.FMParser.MultiplicativeExpression(FMParser.java:754)
    at freemarker.core.FMParser.AdditiveExpression(FMParser.java:706)
    at freemarker.core.FMParser.RangeExpression(FMParser.java:886)
    at freemarker.core.FMParser.RelationalExpression(FMParser.java:834)
    at freemarker.core.FMParser.EqualityExpression(FMParser.java:797)
    at freemarker.core.FMParser.AndExpression(FMParser.java:953)
    at freemarker.core.FMParser.OrExpression(FMParser.java:975)
    at freemarker.core.FMParser.Expression(FMParser.java:534)
    at freemarker.core.FMParser.StringOutput(FMParser.java:1508)
    at freemarker.core.FMParser.MixedContentElements(FMParser.java:3647)
    at freemarker.core.FMParser.List(FMParser.java:1731)
    at freemarker.core.FMParser.FreemarkerDirective(FMParser.java:3329)
    at freemarker.core.FMParser.MixedContentElements(FMParser.java:3697)
    at freemarker.core.FMParser.If(FMParser.java:1611)
    at freemarker.core.FMParser.FreemarkerDirective(FMParser.java:3325)
    at freemarker.core.FMParser.MixedContentElements(FMParser.java:3697)
    at freemarker.core.FMParser.Macro(FMParser.java:2630)
    at freemarker.core.FMParser.FreemarkerDirective(FMParser.java:3352)
    at freemarker.core.FMParser.MixedContentElements(FMParser.java:3697)
    at freemarker.core.FMParser.Root(FMParser.java:4280)
    at freemarker.template.Template.<init>(Template.java:254)
    at freemarker.cache.TemplateCache.loadTemplate(TemplateCache.java:548)
    at freemarker.cache.TemplateCache.getTemplateInternal(TemplateCache.java:438)
    at freemarker.cache.TemplateCache.getTemplate(TemplateCache.java:291)
    at freemarker.template.Configuration.getTemplate(Configuration.java:2438)
    at freemarker.core.Environment$LazilyInitializedNamespace.initialize(Environment.java:2901)
    at freemarker.core.Environment$LazilyInitializedNamespace.ensureInitializedTME(Environment.java:2876)
    ... 66 more

Freemarker's current maintainer's comment is here: http://stackoverflow.com/questions/37298463/freemarker-2-3-24-auto-escape-and-spring-ftl-macros-issue

ref. http://docs.spring.io/spring/docs/current/spring-framework-reference/html/view.html

Here's a reproduce code: https://github.com/tokuhirom/spring-boot-issues-6994

---

Issue Links:

• spring.ftl does not support turning off escaping for some macros on Spring 5 [SPR-16951] #21489 spring.ftl does not support turning off escaping for some macros on Spring 5

Referenced from: commits 72a8868

[spring-projects-issues]

Juergen Hoeller commented

There is nothing we can do about this immediately since the historic versions of FreeMarker - up until half a year ago - do not give us any other option than to use "?html" there, and we need to remain compatible with them. We can only really revise this for Spring Framework 5.0.

For the time being, since spring.ftl is effectively just a regular FreeMarker template that we happen to ship, you could create a customized copy of it and use that instead?

[spring-projects-issues]

Tokuhiro Matsuno commented

I see. I'll copy spring.ftl for now, and waiting spring 5.0 release!

[spring-projects-issues]

Juergen Hoeller commented

We're declaring spring.ftl as #ftl output_format="HTML" now, just no_esc'ing our attributes expressions but otherwise relying on auto-escaping (and therefore requiring FreeMarker 2.3.24+). This allows for application templates to use any format declaration, including a configuration-wide HTML format setting.

[arhamranjha]

Juergen Hoeller commented

We're declaring spring.ftl as #ftl output_format="HTML" now, just no_esc'ing our attributes expressions but otherwise relying on auto-escaping (and therefore requiring FreeMarker 2.3.24+). This allows for application templates to use any format declaration, including a configuration-wide HTML format setting.

@jhoeller So I am using Spring 5.3.xx right now with a simple ftl(i have simplified it for now)

<#import "/spring.ftl" as spring/>
 <@spring.formInput path="${textInputName}" attributes='maxlength="1024"  '/>

with some simple free-marker settings from context.xml

       <property name="freemarkerSettings">
            <props>
                <prop key="datetime_format">yyyy-MM-dd</prop>
                <prop key="url_escaping_charset">UTF-8</prop>
            </props>
        </property>

And i have enable escaping in web.xml

        <context-param>
		<param-name>defaultHtmlEscape</param-name>
		<param-value>true</param-value>
	</context-param>

I am facing an issue with double escaping. A simple value like "abc" in input will be render as "abc" in input value on a page reload.

• One is from spring when it tries to bind status
  

  spring-framework/spring-webmvc/src/main/java/org/springframework/web/servlet/support/BindStatus.java

  Lines 166 to 168 in 08bc1a0

  	if (htmlEscape && this.value instanceof String) {
  	this.value = HtmlUtils.htmlEscape((String) this.value);
  	}

• Other one is from ftl since the spring.ftl uses #ftl output_format="HTML" and that causes autoEscape below to be set to true
  https://github.com/apache/freemarker/blob/df938ce6120ca155b87d48df97b3a6d62123b17f/src/main/java/freemarker/core/DollarVariable.java#L67-L71

Are we suppose to turn off escaping from spring from web.xml (which don't seem safe to me) or is it something else that I missed ?