-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathBlogTestingold.html
325 lines (320 loc) · 19.8 KB
/
BlogTestingold.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
<!DOCTYPE html>
<html>
<head>
<!--
- BlogTesting.htm - Code Artist Thoughts
- ver 1.0 - 02 September 2016
- Jim Fawcett, Syracuse University
-->
<meta http-equiv="content-type" content="text/html;charset=UTF-8" />
<meta name="description" content="Software Engineering course notes. Code Samples. Software Links" />
<meta name="keywords" content="Lecture, Notes, Code, Syracuse,University" />
<meta name="Author" content="Jim Fawcett" />
<meta name="Author" content="James Fawcett" />
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<title>Blog.Testing</title>
<script src="js/ScriptsUtilities.js"></script>
<script src="js/ScriptsTemplate.js"></script>
<script src="js/ScriptsKeyboard.js"></script>
<script src="js/ScriptsMenu.js"></script>
<link rel="stylesheet" href="css/StylesTemplate.css" />
<link rel="stylesheet" href="css/StylesDefault.css" />
<link rel="stylesheet" href="css/StylesMenu.css" />
<link rel="stylesheet" href="css/StylesBrownTheme.css" />
</head>
<body id="github" onload="initializeMenu()">
<navKeys-Container>
<nav-Key id="sKey" onclick="toggleSwipeEvents()">S</nav-Key>
<nav-Key id="rKey" onclick="location.reload()">R</nav-Key>
<nav-Key id="tKey" onclick="scrollPageTop()">T</nav-Key>
<nav-Key id="bKey" onclick="scrollPageBottom()">B</nav-Key>
<nav-Key id="hKey" onclick="helpWin()">H</nav-Key>
<nav-Key id="pKey" onclick="loadPrev()">P</nav-Key>
<nav-Key id="nKey" onclick="loadNext()">N</nav-Key>
</navKeys-Container>
<nav>
<div id="navbar"></div>
</nav>
<a id="Next" href="SummerReading.html">N</a>
<a id="Prev" href="BlogOCD.html">P</a>
<header>
<hgroup id="earthpagetitle">
<h1 id="earthtitle">Code Artistry - Software Testing</h1>
</hgroup>
</header>
<!-- page content -->
<h3>Initial Thoughts:</h3>
There are several types of software tests each with its own specific objectives:
<ol>
<li>
<strong>Construction Tests - run by Developers:</strong><br />
As we build a software package we lay out its structure with declarations of one or more classes.
These are implemented by adding a few lines of code in a class method, adding a test for that method
in a construction test stub<sup>1</sup>, then building, and executing the new test. Doing this continuously allows us
to keep the software always running - at first just doing simple things, but progressively adding more
detail, until the class meets its complete set of obligations. We do that for each class until
our package is complete. Then we repeat with the next package and so on until the project has been completed.
</li>
<li>
<strong>Unit Tests - run by Developers:</strong><br />
It is essential that we make packages, on which other packages depend, as free of errors as is practical.
Any errors in a low-level package imply that the operation of most packages that depend on the flawed
package will also be flawed. A unit test is an eyes-on, debugger walk through that exercises every line
of code, function-by-function, toggles every predicate, and examines the results carefully to ensure correct operation.
These are expensive tests and we often choose not to unit test high level packages, especially
Graphical User Interfaces<sup>2</sup> because their operations should be simple and clear.
</li>
<li>
<strong>Integration Tests - run by Developers and QA:</strong><br />
Integration tests are automated tests that exercise changed or newly introduced
code in the context of the entire project baseline<sup>3</sup>, or some large subset of that.
When we create a new package or modify an existing package the changes we make may break, not only packages
that directly depend on the new version, but also on other packages that are not obviously connected. For
example, a package that minipulates data in a database may introduce changes to the database that break other
code that depend on the database state. This can happen even if the new version has no compilation dependency
relationships with the other package.
Integration tests are aggregated
over the course of developement of a large project and are run frequently to ensure that new code doesn't break
the existing baseline. Each integration test driver (there will be many of them) has, embedded in comments of its
source code, a test description that defines what the test is trying to demonstrate and the steps it uses to do that.
</li>
<li>
<strong>Regression Tests - run by QA:</strong><br />
Regression tests are very similar to Integration tests. The only difference is their purpose. We use
regression testing when porting an application to a new platform or modified platform and when we deploy the
application to another site. Here we run a sequence of tests on the entire baseline to show that it still
functions in the same way it did in its original environment.
</li>
<li>
<strong>Performance and Stress Testing - run by SW Architect, and technical Developers:</strong><br />
Performance tests are run to see if the system is meeting its allocated time and CPU resource budgets. Stress
tests are designed to exceed the required input load capacity and demand more of the platform's resources than
expected. The purpose of these tests is to insure that the system survives unusual loads and resource constraints,
continuing to operate, although it may not satisfy all its obligations in those extreme environments.
</li>
<li>
<strong>Qualification Tests - run by QA under supervision of a Program Manager:</strong><br />
The final demonstration of a large software system, called a Qualification Test, is used to demonstrate that the
system meets all of its obligations to the
customer as laid out in a system specification. This qualification testing is a legalistic, not technical, test.
It should be very organized, detailed, and boring<sup>4</sup>. Qualification testing uses both test automation, like
regression tests, but also uses hands-on probing and demonstrating to the customer the system's operational details.
</li>
</ol>
<h3>Implementing Tests:</h3>
<ol>
<li>
<strong>Construction tests</strong> are implemented manually as you work on the initial construction of a system. Every package has
a main function that is included or excluded from compilation by a compiler directive. We refer to this main function
as a test stub. During construction testing for each package we:
<ul class="details">
<li>
Layout class declarations that are initially empty. This allows us to think about the functionality required
and the language we want to provide for higher level packages to use.
</li>
<li>
Begin populating classes with methods, one-at-a-time. Each method starts with a few lines of code.
</li>
<li>
For each new small addition of code we may add a test in the test stub, and build, and execute. When there are flaws
in our implementation, we know were to look - it is very likely to be in the code we just added.
</li>
<li>
Repeat this process until the package has become fully functional.
</li>
<li>
Repeat all of the steps, above, for each package in the project.
</li>
</ul>
</li>
<li>
<strong>Unit tests</strong> walk through every line of code with a debugger ensuring that every branch has been
activated by test inputs. We do this function-by-function for each class in the package.
This means that we will need to:
<ul class="details">
<li>Do some initial planning for test execution.</li>
<li>Build a test driver that supplies inputs and may log package state and results.</li>
<li>The driver provides inputs that drive the code into each of its possible states<sup>5</sup>.</li>
<li>
For complex functions we may need to accept data stored in files or a database and record results in files or database state.
</li>
<li>Each test driver records, in comments, the resources it uses<sup>6</sup>, e.g., files and databases.</li>
<li>
When unit testing of a package is complete, the test driver is saved in a repository for future use.
For critical packages we may repeat unit tests for every new version.
</li>
</ul>
There are also frameworks like NUnit and JUnit that support testing .Net and Java code, respectively. Developers
call these Unit test frameworks, but the testing they support is a combination of parts of unit testing and integration
testing.
</li>
<li>
<strong>Integration tests</strong> are a fundamental part of continuous software integration and are used by
each developer and Quality Assurance (QA) to test code against the
current project baseline or some significant part of the baseline. These are automated tests for which we need:
<ul class="details">
<li>
A <strong>Test Harness</strong> that loads and executes tests on demand.
</li>
<li>
For each test a dynamic link library (DLL)<sup>7</sup> will be loaded by the Test Harness that contains
a test driver and the code to be tested<sup>8</sup>.
</li>
<li>
The test driver derives from an ITest interface and supplies a factory class with a static method to create instances
of all the classes used by the test.
</li>
<li>
A source of test inputs that may be supplied by the developer or, where feasible, by the test harness.
</li>
<li>
A mechanism for logging test results and possibly internal state values. These logs are named, identify the test
developer, the code tested including version, and are time-date stamped for each test execution.
</li>
<li>
A code <strong>Repository</strong> that holds the current project baseline and can deliver, on demand, DLLs for all the code on which
the tested code depends or may affect. Note that implies the repository has the capability to evaluate calling dependency
relationships.
</li>
<li>
Test requests consist of a message, perhaps in XML format, that lists a series of one or more tests to execute. The
<strong>Test Harness</strong> will execute each request on its own thread, probably using a thread pool, and executes
each test in the request sequentially. A developer is likely to submit test requests with only a few tests while
QA will submit test requests with many tests, perhaps all of the tests that have been defined for a specific subsystem.
</li>
<li>
Test drivers and test resources are stored in the <strong>Repository</strong>, as they will be used many times, some perhaps
thousands of times over the lifetime of a project. Essentially test drivers are part of the software baseline.
</li>
</ul>
<p></p>
Ideally the <strong>Repository</strong> and <strong>Test Harness</strong> are designed to work together.
Existing configuration management systems can be made
to work as repositories in the sense we discuss here, but it is hard to get them to deliver just what is needed for a given
test. More likely they clone massive parts of the baseline and give us many more parts to deal with than needed by each test.
That happens because they don't make dependency graphs available. They just provide clones of named parts. Therefore, the
developer has to figure out what is needed, or, more likely, will just clone big globs of code that hopefully contain all of
the depended upon code. Note that a developer certainly knows the packages on which his code depends directly, but is unlikely
to know all of the dependency decendents of those packages.
<p></p>
For the <strong>Repository</strong>, I prefer a dedicated server that stores each version of each package only once,
has metadata for each package that describes its dependency relationships - thus forming a virtual dependency graph for each
package - and contains useful descriptive information in the metadata to support browsing and analysis. It supplies, on demand,
the entire dependency graph of a named component<sup>9</sup>.
<p></p>
The <strong>Repository</strong> will need to provide build facilities for creating DLLs needed for testing. These will almost certainly
be cached in the Repository baseline. It will also need to support versioning information and ownership policies to determine
who is allowed to checkin changes to any package.
<p></p>
Here's a link that discusses using git for large builds and suggests using Maven to manage project dependencies - Maven was designed
to support Java development:
<a href="http://blogs.atlassian.com/2014/04/git-project-dependencies/">Code Dependencies - pain point with classic config mgrs like git</a>
<p></p>
There are dependency managers for .Net, none of which I've used yet. Here's some links:<br />
<a href="http://fsprojects.github.io/Paket/">Paket - package manager with dependency resolution</a><br />
<a href="http://www.nuget.org/">nuget - package manager - dependency resolution?</a><br />
<a href="http://blog.davidebbo.com/2011/03/using-nuget-without-committing-packages.html">Using nuget with repositories</a>
</li>
<li>
<strong>Regression Tests</strong> are really just the same tests used for integration, but are used for other purposes, as
described in the Initial Thoughts section. Their implementation will not differ from integration tests in any significant way.
</li>
<li>
<strong>Performance Tests</strong> are specialized tests that use the integration test framework with drivers that are
designed specifically for timing and throughput analysis. Instead of running many tests on different parts of the baseline
they run many tests repeatedly on the same code, collecting execution times for each and then building outputs that
provide statistics on execution times and throughput.
</li>
<li>
<strong>Stress tests</strong> are specialized tests that work much like performance tests. For performance testing we
provide inputs and an environment that is specified for the application. For stress testing we provide inputs that
exceed the specified load and modify environments in ways that the application was not expected to tolerate. We look
critically for any error states from which the application can not recover while still running. We also look for
corruption of persistant state in databases or files that will prevent the system from successfully restarting into
a fully functional state.
</li>
<li>
<strong>Qualification Tests</strong> are very thorough examinations of the operations of a system to ensure that it
meets all of its specified obligations. A <strong>Test Harness</strong> and <strong>Repository</strong> can be
excellent resources for Qualification tests.
<p></p>
The extent of Qualification testing is very much dependent on the size of each project and the context in which the
system is built, e.g., large projects funded by government agencies vs. internal industrial projects carried out to
extend existing products or to implement a new product.
</li>
</ol>
<t-b>
You may wish to look at <a href="../CSE681/lectures/Project5-F2016.htm">CSE681 - SMA: Project #5</a> to see how these parts,
<strong>Test Harness</strong> and <strong>Repository</strong>
fit into a Software Development Environment.
</t-b>
<h3>Your use of Testing in Courses:</h3>
<t-b>
In the courses you will take in your program of study, as you work on class projects, you should always use construction
testing, starting with a simple system shell that doesn't do much but does compile and run. You add functionality,
package-by-package, in small pieces, adding construction tests with each small addition of functionality. This way
you keep the system always running and progressively acquire more functionality until you are finished or the Project
due date has arrived.
</t-b>
<t-b>
We discuss test frameworks in CSE681 - Software Modeling and Analysis (SMA), and also briefly in
CSE687 - Object Oriented Design (OOD).
For example, in the Fall of 2016 we are developing, in CSE681 - Software Modeling and Analysis, a test harness that
could be used in a collaboration system for software development.
</t-b>
<t-b>
You probably will not engage in Regression, Performance, Stress, or Qualification testing as part of your program of study, but
may acquire that as on the job training once you are working professionally. We used to do some of that in CSE784 - Software Studio,
but unfortunately we are no longer able to offer that course.
</t-b>
<div style="height:30px;"></div>
<hr />
<ol>
<li>
Each package contains a main function enclosed with compiler directives that allow us to compile for stand-along operation
using the main, or for integration into a larger body of code, without the main. We call this main function a
test stub.
</li>
<li>
Graphical User Interfaces should focus on accepting program data from the user and displaying results. It should do
no other computation. Any computation it needs should be delegated to another package that can be tested effectively
in a test framework.
</li>
<li>
The project baseline is all the code associated with the project's current state. It includes only code that has been
successfully tested and checked into the project's Repository. That includes both product code and test drivers.
</li>
<li>
If a Qualification test is exciting, we have problems. Qualification should proceed in an orderly fashion, demonstrating
to the customer(s) that the system works as specified. We don't want any unexpected behavior or lack of expected behavior.
</li>
<li>
If a function has high complexity, it may be untestable. That is, it may be far easier to throw away the function and
rebuild using a simpler structure than to test all the permutations of its many states.
</li>
<li>
These comments form a test description for the tested code. It would be relatively easy to build an analysis tool that
extracts these "descriptions" for all of the test drivers and build a test document in real time - always up-to-date
and as accurate as the comments in the driver code.
</li>
<li>
A dynamic link library (DLL) is a compiled binary that is constructed to load into an executable image at run-time.
Unix and Linux developers refer to these as "shared" libraries.
</li>
<li>
It would be equally effective to build separate DLLs for the test driver and tested code, where the test driver loads
the tested code DLL when it is loaded.
</li>
<li>
The Repository recursively scans metadata, starting at the node for the requested component, and recursing into all its
descendents, capturing a list of all the filespecs it encounters. This list is what the Test Harness uses to get the
DLLs it needs for any given test. Obviously it would be desirable to cache the downloaded DLLs so we don't keep sending
some of the same components test-after-test.
</li>
</ol>
<p>
<img class="photo" src="Pictures/campusStrip.jpg" alt="Newhouse" style="width:100%;" />
</p>
<info-bar></info-bar>
</body>
</html>