CSE4431 - Systems Validation
& Verification, Quality and Standards
Lectures:
S Ramakrishnan
Semester 2
2005 (July - Nov)
Venue: Bldg.11/Room H6
Time: Wednesday 10am - 12 noon
MUSE (Monash University Software
Engineering) Studio Labs: Bldg. 26, Room G.13
(Check MUSE Lab access facility
arrangements with the Clayton School IT Office)
6 points, Second
Semester,
Clayton
Prerequisites:
CSE2201,
CSE2304, CSE2305, CSE3308, BUS2176 and CSE2/3391/ CSE2/3395
Co-requisite:
CSE4213
Prohibitions: None
Synopsis: This
subject covers
the products, processes, techniques and tools for system validation
&
verifications including acceptance tests. Commercial Testing Tools from
Rational, Mercury Interactive and others will be used to apply in
practice knowledge
learnt about software testing from a theoritical perspective.
Inspection and
testing methodologies, analysis of artifacts, robustness, performance
analysis
configuration management, quality assurance plan and standards
including
ISO9000/AS39000, compliance, assessment, certification issues are
covered. It
shows how to predict, analyse and control defects in complex software
systems.
It introduces verification methods such as inductive methods for safety
properties. It covers operational semantics for sequential
and concurrent programs based on Hoare logic -
assertion mechanisms - precondition, postcondition and invariants with
a view
to systematic test planning and validation.
Objectives:
- Understand
the role of
validation & verification methods in the system life cycle.
- Gain
experience in using
commercial validation tools such as TestStudio from Rational and
TestDirector
from Mercury Interactive, and other similar products to help detect software
system defects. Also gain experience in Performance and Load Testing
with
testing tools from these vendors.
- Appreciate
how assertion
mechanisms impact reasoning.
- Be
able to analyse and
control defects in complex systems.
- Have
an understanding of
inspection & testing methods, configuration management,
performance, and
quality standards issues.
Assessment:
Assignments 80%,
Demonstration 20%
Meeting
Times
Wed. 10am-12 noon
(Venue: Bldg.11, H6)
AV Equipment Supervisor in Menzies Bldg.,
Ground Floor, Bldg.11, H6 :
Mr Scott.Doyle, scott.doyle@its.monash.edu.au,
ph: 53289
Consultation
Days
Mondays
and Wednesdays (Bldg.75, Room 194) - 3 - 4pm
Lectures - Week by week Lecture
overheads, and
required
readings - Under
Construction and
subject
to change; Slides being updated and additional readings
will be
assigned)
Announcements:
Week 5 Aug 15: Discuss Revised Deadlines; Week 3 (Aug 3): Part of the
Lecture session will be devoted to Eclipse,
JUnit & Coverage Tool Demonstration. Students are expected to
attend the session to get to know these tools in action. Required
knowledge for Assignment 1.
Lectures :
Week
by Week Topic overview
Prereading,
Hurdles and Assignments
Assignment details:
Assignment
1, 40 %, Due Date: Week 7, 31st Aug. 2005
Assignment 2,
20%,
Due
Date:
Week
9, Sep 14th 2005
Assignment 3,
40%,
Due
Dates: Week 12, wed 12th Oct 2005,
Presentations: Week 12 & 13
Lecture
|
Date
|
Topic Details
|
Readings
|
1
|
Jul 20
|
Administration
details
Introduction to subject
Assessment Details
Basic
Testing Concepts, IEEE
Definition,
<29
Slides>
Tool-Validation
Compliance - (TVAL-C) framework
TVAL-C
|
NIST
Report - july 2002 (300 + pages)
I V&V
Stephen R.
Schach (1996), Testing:
Principles and Practice, ACM Computing Surveys, 28:1, 277--279 (1)
SWEBOK
KA Description of Software Testing V.1.0 - A. Bertolino, CNR, Italy
shown in terms of SWEBOK
Software Testing KAs
Also refer to http://www.csse.monash.edu.au/~doit
for a mapping of our BSE against SWEBOK.
|
2
|
Jul 27
|
Testing Foundation
Faults
vs. Failures <30
Slides>
|
M Young, R N Taylor
(1989), Rethinking
the taxonomy of fault detection techniques, Proceedings of the
11th International conference on Software engineering
Rethinking
the taxonomy of fault detection techniques (local copy for class discussion)
|
3
4
|
Aug 3
Aug
10
|
Software
Test Automation
Automated
testing: Introduction
<6
Slides>
JUnit
testing
<34
Slides>
Automating
JUnit, Run Tests from Ant
<17
Slides>
More
on Automated testing
<33
Slides>
<
Requisite Pro - 9 Slides>
<TVAL-C-4
Slides>
<Semester
Start Procedures>
|
Kai Hsiung Chang et al (1991), Heuristic Approach to
Test case Generation , ACM Publ., pp. 174-180
M Fewster
& D Graham (1999), Software Test
Automation, Addison-Wesley
E Dustin, J
Raksha, J Paul (1999), Software Test Automation, Addision-Wesley
Automated Testing tools - Rational
Tools from IBM, Mercury
Interactive testing tools,
Compuware
Testing Tools
Eclipse / JUnit / djUnit demonstration in the MUSE Lab
|
4
|
Aug 10
|
Testing
Techniques <17
slides>
Testing
in Practice <Slides>
Documentation
for Tests <12
slides>
Test case Templates from IEEE Std
(for use in Class and not
for commercial use ) and adapated versions on
the web ieeestd-829-1998.pdf
GenericTestPlan.doc
testcase-template.doc
|
E. J. Weyuker
(1988), The evaluation of program-based software test data
adequacy criteria, Communications of the ACM, Vol 31(6)
The
evaluation of program-based software test data
adequacy criteria (local
copy provided for class discussion)
Hong Zhu,
Patrick A.
V. Hall, John H. R. May (1997), Software unit test coverage and adequacy, Vol
29(4), ACM Computing
Surveys, pp.366-427
:
Software
unit test coverage and adequacy (local copy provided for class
discussion) |
5
|
Aug 17
|
Testing
Techniques
Data
flow coverage criteria <
19 Slides>
Testing
& Certification<10
Slides>
Testing
within SQA <12
Slides>
Testing
program components <15
slides>
Test
Adequacy and Coverage
|
Hutchins M, Foster H,
Goradia T, Ostrand T (1994), Experiments on the
effectiveness of
dataflow- and control-flow-based test adequacy criteria , 16th International
Conference Software Engineering (ICSE),
Page(s):
191 –200
Experiments
on the effectiveness of dataflow- and control-flow-based test adequacy
criteria (local copy for class discussion)
|
6
|
Aug 24
|
Test
case Generation
<25+
slides>
Test
Oracles
<31
slides>
Automated GUI Testing
<20
slides>
|
Kuo-Chung Tai (1993), Predicate-based
test generation for
computer programs, 15th
International
Conference Software Engineering (ICSE), pp. 267--276.
Predicate-based
test generation (local copy for class discussion)
Effectively
prioritizing
tests in development environment, ACM publ.,
pp. 97-106. (local copy for class discussion)
<Assignment
2 - available online)
|
7
|
Aug 31
|
Test
Maturity Models (TMM)
<http://www.improveqs.nl/uktmmpa.htm
>
<
http://www.improveqs.nl/>
Software Standards
<12
slides>
Mutation
Testing
<53
Slides>
|
Refer to papers in the relevant
sections in the pdf file.
http://www.csse.monash.edu.au/~sitar/CSE4431-MUSE2002/V-V-Qual-Stds.pdf |
8
|
Sep 7
|
Component
Testing
Automated
Software Engineering ASE 03 Tutorial by Prof. Gao - <29 slides>
Standard
for Software Component Testing - BCS SIGIST 2001<67 page pdf>
Model
Driven Architectures, Model Transformation
Dagstuhl
seminar by Jean Marc Jezequel on Model Transformation
Model
Transformation with a dedicated impertive language
|
http://www.csse.monash.edu.au/~sitar/CSE4431-MUSE2002/V-V-Qual-Stds.pdf |
9
|
Sep 14
|
Interviews - Assignment 1 |
http://www.csse.monash.edu.au/~sitar/CSE4431-MUSE2002/V-V-Qual-Stds.pdf |
10
|
Sep 21
|
Interviews - Assignment 2 |
http://www.csse.monash.edu.au/~sitar/CSE4431-MUSE2002/V-V-Qual-Stds.pdf |
|
|
Semester Break Sep 26- Sep 30th
2005
|
|
11
|
Oct 5
|
Model Driven Development and
Testing
<CD - Inside the UML with IBM Rational from IBM, Sep 2003>
<Towards
Model-Driven
Testing - http://www.elsevier.nl/locate/entcs/volume82.html>
<Model
driven development and
testing - A case study -
http://modeldrivenarchitecture.esi.es/pdf/paper4-3.pdf>
<Model
driven testing in Product family context in ESI>
|
http://www.csse.monash.edu.au/~sitar/CSE4431-MUSE2002/V-V-Qual-Stds.pdf |
12
|
Oct 12
|
Presentation - Research papers
|
http://www.csse.monash.edu.au/~sitar/CSE4431-MUSE2002/V-V-Qual-Stds.pdf |
13
|
Oct 19
|
Presentation - Research papers |
http://www.csse.monash.edu.au/~sitar/CSE4431-MUSE2002/V-V-Qual-Stds.pdf |
Some URLS of interest:
http://www.sqa-test.com/articles.html
http://polaris.umuc.edu/~skerby/help/wbib_swe.htm
http://www.irisa.fr/pampa/themes/themes-e.html
http://www.rbsc.com/pages/who_who.html
http://web.comlab.ox.ac.uk/oucl/research/grants/eh.html
http://www.ercim.org/publication/Ercim_News/enw32/lemetayer.html
http://www.irisa.fr/vertecs/welcome-english.html
http://www.newcastle.research.ec.org/pdcs/book/pubs.html
http://www.research.ibm.com/journal/sj41-1.html
http://www.inf.ethz.ch/personal/meyer/publications/computer/ariane.pdf
http://www.ivv.nasa.gov/
july 2002 report http://www.nist.gov/public_affairs/releases/n02-07.htm
http://www.nist.gov/director/prog-ofc/report02-3.pdf 300page + document
- How to read an Engineering Research Paper from Bill Griswold,
Dept Comp.Sc & Engineering, University California, San
Diego, Standard one page form that he gets
the students to use -
http://www.cs.ucsd.edu/users/wgg/CSE210/paperform.pdf
Articles for
Discussion Topics and Papers of
interest in CSE4431(Updated
regularly)
Refer
to
http://portal.acm.org or google or Hargrave Library for papers
such as
Mary Jean Harrold
(2000), Testing: a road map, Proceedings of
the Conference on The Future of Software Engineering May 2000, pp. 61-82
Communications of the ACM -
Location: Hargrave-Andrew Library Serials Call Number: 510.7806
A849C,
Location: Caulfield Serials Collection Call Number: 001.6405
ASS
And others
listed in the pre-reading section in the table given above and in the
“Selected
Publications List” available at
http://www.csse.monash.edu.au/~sitar/CSE4431-MUSE2002/
V-V-Qual-Stds.pdf
From
www.swebok.org - pdf file - Bourque P,
Dupuis R and Moore J W, The Emerging Consensus on the Software
Engineering Body
of Knowledge,
SEPG 2002, feb 2002, pp.49-56 on Software Testing
ACM Computing Surveys - Title: ACM
computing
surveys. [electronic
resource]. Continues: Computing
surveys 0010-4892
Publisher: New York, N.Y. : Association for Computing
Machinery, c1971- Material type: [electronic
resource]. Linked resources: Full
text available online via ProQuest (restricted access)
OOPSLA
ECOOP
IEEE Transactions on Software Engineering
International Conference on Software
Engineering and …
Some Conferences
happening in Software Quality and Testing
in 2002
http://www.soft.com/QualWeek/QW2002/framemaker.pht (Quality Week Europe March 2002)
http://www.qaiusa.com/conferences/oct2002/index.html style="font-size: 9pt; color:
black;">22nd Annual International
Software Testing Conference
October
21 - 25, 2002 ~ Orlando, FL)
http://www.softdim.com/psqt2002north/
(
International
Conference on
Practical Software Quality Techniques (PSQT)
& International Conference on Practical
Software
Testing Techniques (PSTT)
PSQT/PSTT 2002 North)
Student Paper
Refer
to http://www.csse.monash.edu.au/~sitar/CSE4431-MUSE2002/V-V-Qual-Stds.pdf for topic areas for your Assignment 3.
Choose a topic and ask the lecturer to okay it, as students are
expected to
choose from a variety of topic areas from the pdf file given or a topic
of
special interest to the student in the area of software testing.
You
may
want to look for
topics for your paper and links to active researchers
by looking at some of the premier conference URLS such as for
Automated Software Engineering (ASE), International Symposium on
Software
Testing and Analysis(ISSTA), International Symposium Software
Reliability
Engineering (ISSRE), International Conference Software Engineering
(ICSE) and
associated/colocated workshops in these conferences and symposiums,
which are
often excellent as well), and other conferences and Journals in this
area. You
may find the following URLS useful:
http://www.ase-conference.org/ASE2004/ 19th IEEE International
conference, Linz, Austria, September 20-25, 2004. Some of the workshops
in
conjunction with ASE2004 are:
http://fates.cs.auc.dk/ 4th International workshop on
formal approaches to testing software with ASE04
http://eecs.oregonstate.edu/issta2004/ Boston, Massachusetts, USA, July 11-14,
2004
Some
of the
workshops in
conjunction with International Symposium on Software Testing and
Analysis (ISSTA
2004):
http://www.cs.ucsb.edu/~bultan/tav-web/ Workshop on Testing, Analysis and
Verification of Web Services (TAV-WEB 2004)
with ISSTA 2004
http://www.sce.carleton.ca/squall/WERST2004/ Workshop on Empirical Research in
Software
Testing (WERST'2004) with ISSTA2004
http://www.issre.org/2004/ 15th
IEEE International Symposium Software Reliability Engineering,
Saint-Malo,
Bretagne, France, Nov 2-5, 2004, Theme:
Achieving software dependability through model driven engineering
http://conferences.iee.org/icse2004/ 26th International conference
Software Engineering, May 23-38, Scotland, UK (also refer to
interesting
colocated workshops)
http://www.sce.carleton.ca/squall/ - The Software Engineering Quality
Engineering Laboratory, Carleton University, Ottawa, Canada for some interesting papers
http://www.irisa.fr/triskell/welcome.htm University of Rennes, France
Student Paper
Must include an
abstract, overview of the paper, motivation, literature review,
students'
contribution,
related work,
weakness of the techniques discussed, further work and a
summary/conclusion.
Judging for Best paper and
possible
submission to a conference.
Your paper will
be judged for its content , written style & format and sticking to
deadline
schedules given for submissions.
Content and Style
1. Must include an abstract,
overview of the paper, motivation,
literature review, students' contribution, related work, weakness of
the
techniques discussed, further work and a summary/conclusion
2. Use academic
style writing and do not use sales or marketing or flowery language
3. Check for
typos, grammar,style
4. Use page
numbers in the paper submitted
5. References
must include refereed articles from Proceedings and Journals and not
just from
popular magazines or websites. A
minimum of six references must be included.
6. Books are fine but
cannot be the sole references. Citing must be
done properly. Refer to standards for citation in past
Honours theses and/or in the refereed articles you have
refereed to.
Page
Limit
Paper limit of 10
pages applies for student papers. Use Latex template given
and produce a ps file.
Convert ps to pdf
from Linux using ps2pdf filename
command and submit the paper as a pdf
file.
Presentation
1.
Refer
to
Guidelines given for Presentation by Sita for your CSE4002
Presentation#1
seminar in semester 1.
2.
Stick to time limit given.
3. Practice
talk in front of a mirror at home or with a friend and do not come
unprepared/unrehearsed to your presentation.
4. Attendance
to all your peers' presentation is compulsory. Marks will be deducted
for non
attendance.
5.
Participation
by asking relevant questions at the end of your peer's presentation
carries marks
for you asking the question
and to the presenter if he/she
answers
them clearly. Collusion between
presenter and the peer member re: Q/A carries a heavy penalty.
6.
Follow
a
strict decorum. No whistling or heckling or inappropriate behaviour
will NOT be
tolerated.
7.
Presentations
are to be treated as strictly academic/professional activities.