[ZEPPELIN-1614] Remove markdown4j dep

STYLE.md

@@ -7,7 +7,7 @@ app/styles/looknfeel
 Overall look and theme of the Zeppelin notebook page can be customized here.
 
 ### Code Syntax Highlighting
-There are two parts to code highlighting. First, Zeppelin uses the Ace Editor for its note paragraphs. Color style for this can be changed by setting theme on the editor instance. Second, Zeppelin's Markdown interpreter calls markdown4j to emit HTML, and such content may contain <pre><code> tags that can be consumed by Highlight.js.
+There are two parts to code highlighting. First, Zeppelin uses the Ace Editor for its note paragraphs. Color style for this can be changed by setting theme on the editor instance. Second, Zeppelin's Markdown interpreter calls pegdown parser to emit HTML, and such content may contain <pre><code> tags that can be consumed by Highlight.js.
 
 #### Theme on Ace Editor
 app/scripts/controllers/paragraph.js  
@@ -16,7 +16,7 @@ Call setTheme on the editor with the theme path/name.
 [List of themes on GitHub](https://github.com/ajaxorg/ace/tree/master/lib/ace/theme)
 
 #### Style for Markdown Code Blocks
-Highlight.js parses and converts <pre><code> blocks from markdown4j into keywords and language syntax with proper styles. It also attempts to infer the best fitting language if it is not provided. The visual style can be changed by simply including the desired [stylesheet](https://github.com/components/highlightjs/tree/master/styles) into app/index.html. See the next section on build.
+Highlight.js parses and converts <pre><code> blocks from pegdown parser into keywords and language syntax with proper styles. It also attempts to infer the best fitting language if it is not provided. The visual style can be changed by simply including the desired [stylesheet](https://github.com/components/highlightjs/tree/master/styles) into app/index.html. See the next section on build.
 
 Note that code block background color is overriden in app/styles/notebook.css (look for .paragraph .tableDisplay .hljs).
 

docs/install/upgrade.md

@@ -53,3 +53,4 @@ So, copying `notebook` and `conf` directory should be enough.
  - Mapping from `%jdbc(prefix)` to `%prefix` is no longer available. Instead, you can use %[interpreter alias] with multiple interpreter setttings on GUI.
  - Usage of `ZEPPELIN_PORT` is not supported in ssl mode. Instead use `ZEPPELIN_SSL_PORT` to configure the ssl port. Value from `ZEPPELIN_PORT` is used only when `ZEPPELIN_SSL` is set to `false`.
  - The support on Spark 1.1.x to 1.3.x is deprecated.
+ - From 0.7, we uses `pegdown` as the `markdown.parser.type` option for the `%md` interpreter. Rendered markdown might be different from what you expected

docs/interpreter/markdown.md

@@ -25,14 +25,24 @@ limitations under the License.
 
 ## Overview
 [Markdown](http://daringfireball.net/projects/markdown/) is a plain text formatting syntax designed so that it can be converted to HTML.
-Apache Zeppelin uses [markdown4j](https://github.com/jdcasey/markdown4j) and [pegdown](https://github.com/sirthias/pegdown) as markdown parsers.
+Apache Zeppelin uses [pegdown](https://github.com/sirthias/pegdown) and [markdown4j](https://github.com/jdcasey/markdown4j) as markdown parsers.
 
 In Zeppelin notebook, you can use ` %md ` in the beginning of a paragraph to invoke the Markdown interpreter and generate static html from Markdown plain text.
 
-In Zeppelin, Markdown interpreter is enabled by default and uses the [markdown4j](https://github.com/jdcasey/markdown4j) parser.
+In Zeppelin, Markdown interpreter is enabled by default and uses the [pegdown](https://github.com/sirthias/pegdown) parser.
 
 <img src="../assets/themes/zeppelin/img/docs-img/markdown-interpreter-setting.png" width="60%" />
 
+## Example
+
+The following example demonstrates the basic usage of Markdown in a Zeppelin notebook.
+
+<img src="../assets/themes/zeppelin/img/docs-img/markdown-example.png" width="70%" />
+
+## Mathematical expression
+
+Markdown interpreter leverages %html display system internally. That means you can mix mathematical expressions with markdown syntax. For more information, please see [Mathematical Expression](../displaysystem/basicdisplaysystem.html#mathematical-expressions) section.
+
 ## Configuration
 <table class="table-configuration">
   <tr>
@@ -42,31 +52,25 @@ In Zeppelin, Markdown interpreter is enabled by default and uses the [markdown4j
   </tr>
   <tr>
     <td>markdown.parser.type</td>
-    <td>markdown4j</td>
-    <td>Markdown Parser Type. <br/> Available values: markdown4j, pegdown.</td>
+    <td>pegdown</td>
+    <td>Markdown Parser Type. <br/> Available values: pegdown, markdown4j.</td>
   </tr>
 </table>
 
-## Example
-
-The following example demonstrates the basic usage of Markdown in a Zeppelin notebook.
 
-<img src="../assets/themes/zeppelin/img/docs-img/markdown-example.png" width="70%" />
+### Pegdown Parser
 
-## Mathematical expression
+`pegdown` parser provides github flavored markdown.
 
-Markdown interpreter leverages %html display system internally. That means you can mix mathematical expressions with markdown syntax. For more information, please see [Mathematical Expression](../displaysystem/basicdisplaysystem.html#mathematical-expressions) section.
+<img src="../assets/themes/zeppelin/img/docs-img/markdown-example-pegdown-parser.png" width="70%" />
 
+`pegdown` parser provides [YUML](http://yuml.me/) and [Websequence](https://www.websequencediagrams.com/) plugins also. 
 
+<img src="../assets/themes/zeppelin/img/docs-img/markdown-example-pegdown-parser-plugins.png" width="70%" />
 
 ### Markdown4j Parser
 
-`markdown4j` parser provides [YUML](http://yuml.me/) and [Websequence](https://www.websequencediagrams.com/) extensions 
-
-<img src="../assets/themes/zeppelin/img/docs-img/markdown-example-markdown4j-parser.png" width="70%" />
-
-### Pegdown Parser
+Since pegdown parser is more accurate and provides much more markdown syntax  
+`markdown4j` option might be removed later. But keep this parser for the backward compatibility.
 
-`pegdown` parser provides github flavored markdown.
 
-<img src="../assets/themes/zeppelin/img/docs-img/markdown-example-pegdown-parser.png" width="70%" />

markdown/src/main/java/org/apache/zeppelin/markdown/Markdown.java

@@ -33,13 +33,17 @@
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-/** MarkdownInterpreter interpreter for Zeppelin. */
+/**
+ * MarkdownInterpreter interpreter for Zeppelin.
+ */
 public class Markdown extends Interpreter {
   private static final Logger LOGGER = LoggerFactory.getLogger(Markdown.class);
 
   private MarkdownParser parser;
 
-  /** Markdown Parser Type. */
+  /**
+   * Markdown Parser Type.
+   */
   public enum MarkdownParserType {
     PEGDOWN {
       @Override
@@ -82,7 +86,8 @@ public void open() {
   }
 
   @Override
-  public void close() {}
+  public void close() {
+  }
 
   @Override
   public InterpreterResult interpret(String markdownText, InterpreterContext interpreterContext) {
@@ -99,7 +104,8 @@ public InterpreterResult interpret(String markdownText, InterpreterContext inter
   }
 
   @Override
-  public void cancel(InterpreterContext context) {}
+  public void cancel(InterpreterContext context) {
+  }
 
   @Override
   public FormType getFormType() {

markdown/src/main/java/org/apache/zeppelin/markdown/Markdown4jParser.java

@@ -23,7 +23,9 @@
 
 import java.io.IOException;
 
-/** Markdown Parser using markdown4j processor . */
+/**
+ * Markdown Parser using markdown4j processor.
+ */
 public class Markdown4jParser implements MarkdownParser {
   private Markdown4jProcessor processor;
 

markdown/src/main/java/org/apache/zeppelin/markdown/MarkdownParser.java

@@ -17,7 +17,9 @@
 
 package org.apache.zeppelin.markdown;
 
-/** Abstract Markdown Parser. */
+/**
+ * Abstract Markdown Parser.
+ */
 public interface MarkdownParser {
   String render(String markdownText);
 }

markdown/src/main/java/org/apache/zeppelin/markdown/ParamVar.java

@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.markdown;
+
+import org.parboiled.support.Var;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Implementation of Var to support parameter parsing.
+ *
+ * @param <K> Key
+ * @param <V> Value
+ */
+public class ParamVar<K, V> extends Var<Map<K, V>> {
+
+  public ParamVar() {
+    super(new HashMap<K, V>());
+  }
+
+  public boolean put(K key, V value) {
+    get().put(key, value);
+    return true;
+  }
+}

markdown/src/main/java/org/apache/zeppelin/markdown/PegdownParser.java

@@ -19,17 +19,23 @@
 
 import org.pegdown.Extensions;
 import org.pegdown.PegDownProcessor;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.pegdown.plugins.PegDownPlugins;
 
-/** Markdown Parser using pegdown processor. */
+/**
+ * Markdown Parser using pegdown processor.
+ */
 public class PegdownParser implements MarkdownParser {
   private PegDownProcessor processor;
 
+  public static final long PARSING_TIMEOUT_AS_MILLIS = 5000;
+  public static final int OPTIONS = Extensions.ALL_WITH_OPTIONALS - Extensions.ANCHORLINKS;
+
   public PegdownParser() {
-    int pegdownOptions = Extensions.ALL_WITH_OPTIONALS - Extensions.ANCHORLINKS;
-    int parsingTimeoutAsMillis = 5000;
-    processor = new PegDownProcessor(pegdownOptions, parsingTimeoutAsMillis);
+    PegDownPlugins plugins = new PegDownPlugins.Builder()
+        .withPlugin(PegdownYumlPlugin.class)
+        .withPlugin(PegdownWebSequencelPlugin.class)
+        .build();
+    processor = new PegDownProcessor(OPTIONS, PARSING_TIMEOUT_AS_MILLIS, plugins);
   }
 
   @Override
@@ -45,12 +51,14 @@ public String render(String markdownText) {
     return html;
   }
 
-  /** wrap with markdown class div to styling DOM using css. */
+  /**
+   * wrap with markdown class div to styling DOM using css.
+   */
   public static String wrapWithMarkdownClassDiv(String html) {
     return new StringBuilder()
-      .append("<div class=\"markdown-body\">\n")
-      .append(html)
-      .append("\n</div>")
-      .toString();
+        .append("<div class=\"markdown-body\">\n")
+        .append(html)
+        .append("\n</div>")
+        .toString();
   }
 }

markdown/src/main/java/org/apache/zeppelin/markdown/PegdownWebSequencelPlugin.java

@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.markdown;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.parboiled.BaseParser;
+import org.parboiled.Rule;
+import org.parboiled.support.StringBuilderVar;
+import org.pegdown.Parser;
+import org.pegdown.ast.ExpImageNode;
+import org.pegdown.ast.TextNode;
+import org.pegdown.plugins.BlockPluginParser;
+import org.pegdown.plugins.PegDownPlugins;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.OutputStreamWriter;
+import java.net.URL;
+import java.net.URLConnection;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+
+/**
+ * Pegdown plugin for Websequence diagram
+ */
+public class PegdownWebSequencelPlugin extends Parser implements BlockPluginParser {
+
+  private static final String WEBSEQ_URL = "http://www.websequencediagrams.com";
+
+  public PegdownWebSequencelPlugin() {
+    super(PegdownParser.OPTIONS,
+        PegdownParser.PARSING_TIMEOUT_AS_MILLIS,
+        DefaultParseRunnerProvider);
+  }
+
+  public PegdownWebSequencelPlugin(Integer opts, Long millis, ParseRunnerProvider provider,
+                                   PegDownPlugins plugins) {
+    super(opts, millis, provider, plugins);
+  }
+
+  public static final String TAG = "%%%";
+
+  Rule StartMarker() {
+    return Sequence(Spn1(), TAG, Sp(), "sequence", Sp());
+  }
+
+  String EndMarker() {
+    return TAG;
+  }
+
+  Rule Body() {
+    return OneOrMore(TestNot(TAG), BaseParser.ANY);
+  }
+
+  Rule BlockRule() {
+    StringBuilderVar style = new StringBuilderVar();
+    StringBuilderVar body = new StringBuilderVar();
+
+    return NodeSequence(
+        StartMarker(),
+        Optional(
+            String("style="),
+            Sequence(OneOrMore(Letter()), style.append(match()), Spn1())),
+        Sequence(Body(), body.append(match())),
+        EndMarker(),
+        push(
+            new ExpImageNode("title",
+                createWebsequenceUrl(style.getString(), body.getString()),
+                new TextNode("")))
+    );
+  }
+
+  public static String createWebsequenceUrl(String style,
+                                            String content) {
+
+    style = StringUtils.defaultString(style, "default");
+
+    OutputStreamWriter writer = null;
+    BufferedReader reader = null;
+
+    String webSeqUrl = "";
+
+    try {
+      String query = new StringBuilder()
+          .append("style=")
+          .append(style)
+          .append("&message=")
+          .append(URLEncoder.encode(content, "UTF-8"))
+          .append("&apiVersion=1")
+          .toString();
+
+      URL url = new URL(WEBSEQ_URL);
+      URLConnection conn = url.openConnection();
+      conn.setDoOutput(true);
+      writer = new OutputStreamWriter(conn.getOutputStream(), StandardCharsets.UTF_8);
+      writer.write(query);
+      writer.flush();
+
+      StringBuilder response = new StringBuilder();
+      reader = new BufferedReader(
+          new InputStreamReader(conn.getInputStream(), StandardCharsets.UTF_8));
+      String line;
+      while ((line = reader.readLine()) != null) {
+        response.append(line);
+      }
+
+      writer.close();
+      reader.close();
+
+      String json = response.toString();
+
+      int start = json.indexOf("?png=");
+      int end = json.indexOf("\"", start);
+
+      if (start != -1 && end != -1) {
+        webSeqUrl = WEBSEQ_URL + "/" + json.substring(start, end);
+      }
+    } catch (IOException e) {
+      throw new RuntimeException("Failed to get proper response from websequencediagrams.com", e);
+    } finally {
+      IOUtils.closeQuietly(writer);
+      IOUtils.closeQuietly(reader);
+    }
+
+    return webSeqUrl;
+  }
+
+  @Override
+  public Rule[] blockPluginRules() {
+    return new Rule[]{BlockRule()};
+  }
+}