-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathBlogTesting.html
451 lines (444 loc) · 27.8 KB
/
BlogTesting.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
<!DOCTYPE html>
<html>
<!--
BlogTesting.html
-->
<head>
<title>Blog Testing</title>
<meta name="viewport" content="width=device-width, initial-scale=1" />
<!-- <link rel="icon" type="image/x-icon" href="./images/favicon.ico" /> -->
<link rel="stylesheet" href="css/StylesPhoto.css" />
<link rel="stylesheet" href="css/StylesSizerComp.css" />
<!-- PageFrame infrastructure -->
<link rel="stylesheet" href="css/StylesPageFrameDefaults.css" />
<link rel="stylesheet" href="css/StylesPageFrameStructure.css" />
<link rel="stylesheet" href="css/StylesPageFrameMenus.css" />
<link rel="stylesheet" href="css/StylesPageFrameThemePython.css" />
<link rel="stylesheet" href="css/StylesWebComponents.css" />
<script src="js/ScriptsWebComponents.js"></script>
<!--<script src="js/ScriptsPageFrameDefaults.js"></script>-->
<script src="js/ScriptsPageFramePosts.js"></script>
<script src="js/ScriptsPageFramePagesPosts.js"></script>
<script src="js/ScriptsPageFrameKeyboard.js"></script>
<!-- No need for Pages script for pages with no next or prev pages -->
<!--<script src="js/ScriptsPageFramePages.js"></script>-->
<!-- <script src="js/ScriptsTemplate.js"></script>
<link rel="stylesheet" href="css/StylesTemplate.css" /> -->
<style>
h3 {
margin-top: 1.5em;
}
#subtitle {
margin-top: 0.4em;
margin-bottom: 0.3em;
}
#github header summary {
border: 1px solid var(--light);
}
#github summary {
padding-right: 2em;
}
/* #github .menuHead {
margin:0em -0.25em 0.0em -0.25em;
padding:0.25em 0.5em;
} */
</style>
<script>
function load() {
initialize();
//loadif();
}
</script>
<style>
#github note {
display: block;
width:max-content;
border:1px solid red;
padding:0.5em 1.0em;
margin:0.5em 0em;
}
#github .bargraph {
border: 1px solid var(--dark);
/* background-color: #bbb; */
padding: 0.1em 0.5em;
font-size:0.9em;
}
#github table {
border:2px solid var(--dark);
}
#github table td {
padding:0.25em 1.0em;
border:none;
}
body {
user-select:none;
}
</style>
<script>
function clickstat() {
// prevent parent click event handling
event.stopImmediatePropagation();
}
</script>
</head>
<body id="github" onload="load()" style="position:relative;">
<a id="Next" href="BlogStructure.html">Next</a>
<a id="Prev" href="BlogOCD.html">Prev</a>
<page-frame>
<frame-header>
<nav id="navbar"></nav>
</frame-header>
<main id="main">
<div id="about" onclick="this.style.display = 'none'">about</div>
<div id="page">Blog: Testing</div>
<div id="modified">11/26/2024</div>
<div id="hlp"></div>
<a id="top"></a>
<content style="height:100vh; position:relative;">
<header style="cursor:pointer;" onclick="loadif()">
<!-- <a target="_blank" class="repoLink" href="https://github.com/JimFawcett">github Repositories</a> -->
<hgroup id="pagetitle" style="border: 2px solid var(--dark);">
<h1 id="title">Blog: Testing</h1>
<h3 class="indent" id="subtitle">
construction, unit, integration, regression, performance, and qualification
</h3>
</hgroup>
<!-- <img style="width:100%; margin:-0.1em 0em; border:2px solid var(--dark); padding:0.5em; background-color:var(--light);" src="Pictures/officestrip3a.svg" /> -->
<div class="darkItem" onclick="loadif()" style="cursor:pointer; position:relative; padding:0.0em 0em 0.25em 0em; margin-top:-0.50em; border:2px solid var(--dark);">
<a class="repoLinks" target="_blank" href="https://github.com/JimFawcett" style="color:var(--atten); margin-left:1.5em;">About</a>
<div style="font-size:0.9em; position:absolute; top:0.1em; right:1.5em;">click to toggle Site Explorer</div>
<div style="height:0.5em;"></div>
</div>
</header>
<h3>Initial Thoughts:</h3>
Much of this discussion focuses on testing large software baselines developed by several teams.
However, most of the ideas and test types, cited here, are relevent for smaller projects as well, with some
omissions.
<div style="height:0.75em;"></div>
There are several types of software tests each with its own specific objectives:
<ol>
<li id="construction">
<strong>Construction Tests - run by Developers:</strong><br />
<div style="height:0.75em;"></div>
As we build a software package we lay out its structure with declarations of one or more classes.
These are implemented by adding a few lines of code in a class method, adding a test for that method
in a construction test stub<sup>1</sup>, then building, and executing the new test. Doing this continuously allows us
to keep the software always running - at first just doing simple things, but progressively adding more
detail, until the class meets its complete set of obligations. We do that for each class until
our package is complete. Then we repeat with the next package and so on until the project has been completed.
</li>
<li id="unit">
<strong>Unit Tests - run by Developers:</strong><br />
<div style="height:0.75em;"></div>
It is essential that we make packages, on which other packages depend, as free of errors as is practical.
Any errors in a low-level package imply that the operation of most packages that depend on the flawed
package will also be flawed.
<div style="height:0.75em;"></div>
A unit test is an eyes-on, debugger walk through that exercises every line
of code, function-by-function, toggles every predicate, and examines the results carefully to ensure correct operation.
These are expensive tests and we often choose not to unit test high level packages, especially
Graphical User Interfaces<sup>2</sup> because their operations should be simple and clear.
</li>
<li id="integration">
<strong>Integration Tests - run by Developers and QA:</strong><br />
<div style="height:0.75em;"></div>
Integration tests are automated tests that exercise changed or newly introduced
code in the context of the entire project baseline<sup>3</sup>, or some large subset of that.
<div style="height:0.75em;"></div>
When we create a new package or modify an existing package the changes we make may break, not only packages
that directly depend on the new version, but also on other packages that are not obviously connected. For
example, a package that minipulates data in a database may introduce changes to the database that break other
code that depend on the database state. This can happen even if the new version has no compilation dependency
relationships with the other package.
<div style="height:0.75em;"></div>
Integration tests are aggregated
over the course of developement of a large project and are run frequently to ensure that new code doesn't break
the existing baseline. Each integration test driver (there will be many of them) has, embedded in comments of its
source code, a test description that defines what the test is trying to demonstrate and the steps it uses to do that.
</li>
<li id="regression">
<strong>Regression Tests - run by QA:</strong><br />
<div style="height:0.75em;"></div>
Regression tests are very similar to Integration tests. The only difference is their purpose. We use
regression testing when porting an application to a new platform or modified platform and when we deploy the
application to another site. Here we run a sequence of tests on the entire baseline to show that it still
functions in the same way it did in its original environment.
</li>
<li id="performance">
<strong>Performance and Stress Testing - run by SW Architect, and technical Developers:</strong><br />
<div style="height:0.75em;"></div>
Performance tests are run to see if the system is meeting its allocated time and CPU resource budgets. Stress
tests are designed to exceed the required input load capacity and demand more of the platform's resources than
expected. The purpose of these tests is to insure that the system survives unusual loads and resource constraints,
continuing to operate, although it may not satisfy all its obligations in those extreme environments.
</li>
<li id="qualification">
<strong>Qualification Tests - run by QA under supervision of a Program Manager:</strong><br />
<div style="height:0.75em;"></div>
The final demonstration of a large software system, called a Qualification Test, is used to demonstrate that the
system meets all of its obligations to the
customer as laid out in a system specification. This qualification testing is a legalistic, not technical, test.
It should be very organized, detailed, and boring<sup>4</sup>. Qualification testing uses both test automation, like
regression tests, but also uses hands-on probing and demonstrating to the customer the system's operational details.
</li>
</ol>
<h3>Implementing Tests:</h3>
<ol>
<li id="implconstr">
<strong>Construction tests</strong> are implemented manually as you work on the initial construction of a system. Every package has
a main function that is included or excluded from compilation by a compiler directive. We refer to this main function
as a test stub. During construction testing for each package we:
<ul class="details" style="list-style-type: disc;">
<li>
Layout class declarations that are initially empty. This allows us to think about the functionality required
and the language we want to provide for higher level packages to use.
</li>
<li>
Begin populating classes with methods, one-at-a-time. Each method starts with a few lines of code.
</li>
<li>
For each new small addition of code we may add a test in the test stub, and build, and execute. When there are flaws
in our implementation, we know were to look - it is very likely to be in the code we just added.
</li>
<li>
Repeat this process until the package has become fully functional.
</li>
<li>
Repeat all of the steps, above, for each package in the project.
</li>
</ul>
</li>
<li id="implunit">
<strong>Unit tests</strong> walk through every line of code with a debugger ensuring that every branch has been
activated by test inputs. We do this function-by-function for each class in the package.
This means that we will need to:
<ul class="details" style="list-style-type: disc;">
<li>Do some initial planning for test execution.</li>
<li>Build a test driver that supplies inputs and may log package state and results.</li>
<li>The driver provides inputs that drive the code into each of its possible states<sup>5</sup>.</li>
<li>
For complex functions we may need to accept data stored in files or a database and record results in files or database state.
</li>
<li>Each test driver records, in comments, the resources it uses<sup>6</sup>, e.g., files and databases.</li>
<li>
When unit testing of a package is complete, the test driver is saved in a repository for future use.
For critical packages we may repeat unit tests for every new version.
</li>
</ul>
There are also frameworks like NUnit and JUnit that support testing .Net and Java code, respectively. Developers
call these Unit test frameworks, but the testing they support is a combination of parts of unit testing and integration
testing.
</li>
<li id="implintegra">
<strong>Integration tests</strong> are a fundamental part of continuous software integration and are used by
each developer and Quality Assurance (QA) to test code against the
current project baseline or some significant part of the baseline. These are automated tests for which we need:
<ul class="details" style="list-style-type: disc;">
<li>
A <strong>Test Harness</strong> that loads and executes tests on demand.
</li>
<li>
For each test a dynamic link library (DLL)<sup>7</sup> will be loaded by the Test Harness that contains
a test driver and the code to be tested<sup>8</sup>.
</li>
<li>
The test driver derives from an ITest interface and supplies a factory class with a static method to create instances
of all the classes used by the test.
</li>
<li>
A source of test inputs that may be supplied by the developer or, where feasible, by the test harness.
</li>
<li>
A mechanism for logging test results and possibly internal state values. These logs are named, identify the test
developer, the code tested including version, and are time-date stamped for each test execution.
</li>
<li>
A code <strong>Repository</strong> that holds the current project baseline and can deliver, on demand, DLLs for all the code on which
the tested code depends or may affect. Note that implies the repository has the capability to evaluate calling dependency
relationships.
</li>
<li>
Test requests consist of a message, perhaps in XML format, that lists a series of one or more tests to execute. The
<strong>Test Harness</strong> will execute each request on its own thread, probably using a thread pool, and executes
each test in the request sequentially. A developer is likely to submit test requests with only a few tests while
QA will submit test requests with many tests, perhaps all of the tests that have been defined for a specific subsystem.
</li>
<li>
Test drivers and test resources are stored in the <strong>Repository</strong>, as they will be used many times, some perhaps
thousands of times over the lifetime of a project. Essentially test drivers are part of the software baseline.
</li>
</ul>
<div style="height:0.75em;"></div>
Ideally the <strong>Repository</strong> and <strong>Test Harness</strong> are designed to work together.
Existing configuration management systems can be made
to work as repositories in the sense we discuss here, but it is hard to get them to deliver just what is needed for a given
test. More likely they clone massive parts of the baseline and give us many more parts to deal with than needed by each test.
That happens because they don't make dependency graphs available. They just provide clones of named parts. Therefore, the
developer has to figure out what is needed, or, more likely, will just clone big globs of code that hopefully contain all of
the depended upon code. Note that a developer certainly knows the packages on which his code depends directly, but is unlikely
to know all of the dependency decendents of those packages.
<div style="height:0.75em;"></div>
For the <strong>Repository</strong>, I prefer a dedicated server that stores each version of each package only once,
has metadata for each package that describes its dependency relationships - thus forming a virtual dependency graph for each
package - and contains useful descriptive information in the metadata to support browsing and analysis. It supplies, on demand,
the entire dependency graph of a named component<sup>9</sup>.
<div style="height:0.75em;"></div>
The <strong>Repository</strong> will need to provide build facilities for creating DLLs needed for testing. These will almost certainly
be cached in the Repository baseline. It will also need to support versioning information and ownership policies to determine
who is allowed to checkin changes to any package.
<div style="height:0.75em;"></div>
Here's a link that discusses using git for large builds and suggests using Maven to manage project dependencies - Maven was designed
to support Java development:
<a href="http://blogs.atlassian.com/2014/04/git-project-dependencies/">Code Dependencies - pain point with classic config mgrs like git</a>
<div style="height:0.75em;"></div>
There are dependency managers for .Net, none of which I've used yet. Here's some links:<br />
<a href="http://fsprojects.github.io/Paket/">Paket - package manager with dependency resolution</a><br />
<a href="http://www.nuget.org/">nuget - package manager - dependency resolution?</a><br />
<a href="http://blog.davidebbo.com/2011/03/using-nuget-without-committing-packages.html">Using nuget with repositories</a>
</li>
<li id="implregress">
<strong>Regression Tests</strong> are really just the same tests used for integration, but are used for other purposes, as
described in the Initial Thoughts section. Their implementation will not differ from integration tests in any significant way.
</li>
<li id="implperform">
<strong>Performance Tests</strong> are specialized tests that use the integration test framework with drivers that are
designed specifically for timing and throughput analysis. Instead of running many tests on different parts of the baseline
they run many tests repeatedly on the same code, collecting execution times for each and then building outputs that
provide statistics on execution times and throughput.
</li>
<li id="implstress">
<strong>Stress tests</strong> are specialized tests that work much like performance tests. For performance testing we
provide inputs and an environment that is specified for the application. For stress testing we provide inputs that
exceed the specified load and modify environments in ways that the application was not expected to tolerate. We look
critically for any error states from which the application can not recover while still running. We also look for
corruption of persistant state in databases or files that will prevent the system from successfully restarting into
a fully functional state.
</li>
<li id="implqual">
<strong>Qualification Tests</strong> are very thorough examinations of the operations of a system to ensure that it
meets all of its specified obligations. A <strong>Test Harness</strong> and <strong>Repository</strong> can be
excellent resources for Qualification tests.
<div style="height:0.75em;"></div>
The extent of Qualification testing is very much dependent on the size of each project and the context in which the
system is built, e.g., large projects funded by government agencies vs. internal industrial projects carried out to
extend existing products or to implement a new product.
</li>
</ol>
<t-b>
You may wish to look at <a href="../CSE681/lectures/Project5-F2016.htm">CSE681 - SMA: Project #5</a> to see how these parts,
<strong>Test Harness</strong> and <strong>Repository</strong>
fit into a Software Development Environment.
</t-b>
<h3>Your use of Testing in Courses:</h3>
<t-b>
In the courses you will take in your program of study, as you work on class projects, you should always use construction
testing, starting with a simple system shell that doesn't do much but does compile and run. You add functionality,
package-by-package, in small pieces, adding construction tests with each small addition of functionality. This way
you keep the system always running and progressively acquire more functionality until you are finished or the Project
due date has arrived.
</t-b>
<t-b>
We discuss test frameworks in CSE681 - Software Modeling and Analysis (SMA), and also briefly in
CSE687 - Object Oriented Design (OOD).
For example, in the Fall of 2016 we are developing, in CSE681 - Software Modeling and Analysis, a test harness that
could be used in a collaboration system for software development.
</t-b>
<t-b>
You probably will not engage in Regression, Performance, Stress, or Qualification testing as part of your program of study, but
may acquire that as on the job training once you are working professionally. We used to do some of that in CSE784 - Software Studio,
but unfortunately we are no longer able to offer that course.
</t-b>
<div style="height:30px;"></div>
<hr />
<ol>
<li>
Each package contains a main function enclosed with compiler directives that allow us to compile for stand-along operation
using the main, or for integration into a larger body of code, without the main. We call this main function a
test stub.
</li>
<li>
Graphical User Interfaces should focus on accepting program data from the user and displaying results. It should do
no other computation. Any computation it needs should be delegated to another package that can be tested effectively
in a test framework.
</li>
<li>
The project baseline is all the code associated with the project's current state. It includes only code that has been
successfully tested and checked into the project's Repository. That includes both product code and test drivers.
</li>
<li>
If a Qualification test is exciting, we have problems. Qualification should proceed in an orderly fashion, demonstrating
to the customer(s) that the system works as specified. We don't want any unexpected behavior or lack of expected behavior.
</li>
<li>
If a function has high complexity, it may be untestable. That is, it may be far easier to throw away the function and
rebuild using a simpler structure than to test all the permutations of its many states.
</li>
<li>
These comments form a test description for the tested code. It would be relatively easy to build an analysis tool that
extracts these "descriptions" for all of the test drivers and build a test document in real time - always up-to-date
and as accurate as the comments in the driver code.
</li>
<li>
A dynamic link library (DLL) is a compiled binary that is constructed to load into an executable image at run-time.
Unix and Linux developers refer to these as "shared" libraries.
</li>
<li>
It would be equally effective to build separate DLLs for the test driver and tested code, where the test driver loads
the tested code DLL when it is loaded.
</li>
<li>
The Repository recursively scans metadata, starting at the node for the requested component, and recursing into all its
descendents, capturing a list of all the filespecs it encounters. This list is what the Test Harness uses to get the
DLLs it needs for any given test. Obviously it would be desirable to cache the downloaded DLLs so we don't keep sending
some of the same components test-after-test.
</li>
</ol>
<div style="height:1em;"></div>
<p>
<img class="photo" src="Pictures/campusStrip.jpg" alt="Newhouse" style="width:100%;" />
</p>
<div style="height:1em;"></div>
<a id="bottom"></a>
</content>
<page-TOC id="pages" style="display:none;">
</page-TOC>
<page-sections id="sections" style="display:none;">
<menu-elem style="width:0.0em"> </menu-elem>
<menu-elem class="secElem"><a href="#bottom">bottom</a></menu-elem>
<menu-elem class="secElem"><a href="#implqual">impl_qual</a></menu-elem>
<menu-elem class="secElem"><a href="#implstress">impl_stress</a></menu-elem>
<menu-elem class="secElem"><a href="#implperform">impl_perform</a></menu-elem>
<menu-elem class="secElem"><a href="#implregress">impl_regress</a></menu-elem>
<menu-elem class="secElem"><a href="#implintegra">impl_integra</a></menu-elem>
<menu-elem class="secElem"><a href="#implunit">impl_unit</a></menu-elem>
<menu-elem class="secElem"><a href="#implconstr">impl_constr</a></menu-elem>
<menu-elem class="secElem"><a href="#qualification">qualification</a></menu-elem>
<menu-elem class="secElem"><a href="#performance">performance</a></menu-elem>
<menu-elem class="secElem"><a href="#regression">regression</a></menu-elem>
<menu-elem class="secElem"><a href="#integration">integration</a></menu-elem>
<menu-elem class="secElem"><a href="#unit">unit</a></menu-elem>
<menu-elem class="secElem"><a href="#construction">construction</a></menu-elem>
<menu-elem class="secElem"><a href="#top">top</a></menu-elem>
<div class='darkItem popupHeader' style="padding:0.25em 2.0em;" onclick="this.parentElement.style.display='none'">Sections</div>
</page-sections>
</main>
<frame-footer>
<menu-item style="width:2.0em;"> </menu-item>
<menu-elem id="nextLink2" onclick="bottomMenu.next()">Next</menu-elem>
<menu-elem id="prevLink2" onclick="bottomMenu.prev()">Prev</menu-elem>
<menu-elem id="pgbtn" onclick="bottomMenu.pages()">Pages</menu-elem>
<menu-elem onclick="bottomMenu.sections()">Sections</menu-elem>
<menu-elem onclick="bottomMenu.about()">About</menu-elem>
<menu-elem id="kysbtn" onclick="storyHlpMenu.keys()">Keys</menu-elem>
<menu-elem style="margin-right:1em">
<span id="loc" style="display:inline-block; font-weight:normal"></span>
</menu-elem>
</frame-footer>
</page-frame>
<script>
let loc = document.getElementById("loc");
let fn = window.location.href.split(/\/|\\/).pop();
loc.innerHTML = fn + ":";
</script>
</body>
</html>