discussion.tex

\section{Discussion} \label{Sec:discussion}
In this section, we discuss some of the limitations of our current implementation and threats to validity of our results.

\subsection{Limitations}

\headbf{Direct testability of functions} Using our technique, it is not possible to produce test cases directly for all \javascript functions.
Non-testable \javascript functions include (1) anonymous functions, (2) private function closures, and (3) functions that uses the \code{this} keyword as a reference to the object of which that function is a property/method. Depending on where the function is called from, the value of \code{this} can be different.
While these functions can be called at the highest program scope they belong to, it is not possible to call them directly in test cases, which makes it challenging to assess their outcomes.\footnote{This is similar to testing private methods in e.g., Java} This is especially so if the tester is interested in stand-alone examination of these functions.  Based on our observations during the evaluation, there is a non-trivial amount of non-testable \javascript code out there. One possible future direction is to automatically refactor non-testable \javascript code to make it testable. 
 
\headbf{Undetected faults} As mentioned in \secref{results}, our test cases miss some of the faults \ie result in false negatives. 
BunnyHunt (ID 5) and TuduList (ID 12) achieve the lowest fault detection recall, which relates to their low coverage compared to the other applications. 
We describe possible reasons behind the observed low coverage for these applications, and possible ways to alleviate them.  
\begin{enumerate}
\item \textit{Unstable Behaviour of Test Cases:} Testing the correct behaviour of a function requires stable assertions, meaning that
 the test case and its assertions should always be deterministic. 
Thus, to achieve stability we need to remove, (1)  random generation functions as well as functions that rely on the output of a randomized function, and (2) functions or event sequences that exhibit unstable behaviour in different executions due to highly dynamic nature of \javascript. 
Traditional event based testing is not affected by such unstable behavioural characteristics of the functions as they run the events and compare the whole captured state after each event is triggered. One way to alleviate this would be to come up with invariants that characterize these functions and events, and use them for testing.
%However, by removing this set of functions and events from the generated unit and DOM-level tests, the statement coverage is also negatively affected depending on the degree at which these functions/events exist in the application.

\item \textit{Other limitations:} Our implementation is currently not able to trigger events that depend on the keyboard movements or drag and drop actions. 
Therefore, we obtain lower coverage for applications that contain such events due to the presence of functions that are either sensitive to keyboard movements or executed during drag and drop actions. This can be alleviated by a more comprehensive implementation - this is not a fundamental issue.
\end{enumerate}  

\input{threatsToValidity}