CSE4431 - Systems Validation & Verification, Quality and Standards

Lectures: S Ramakrishnan

Semester 2 2005 (July - Nov)

Venue: Bldg.11/Room H6

Time: Wednesday 10am - 12 noon

MUSE (Monash University Software Engineering) Studio Labs: Bldg. 26, Room G.13
 (Check MUSE Lab access facility arrangements  with the Clayton School IT Office)

6 points, Second Semester, Clayton

Prerequisites: CSE2201, CSE2304, CSE2305, CSE3308, BUS2176 and CSE2/3391/ CSE2/3395

Co-requisite: CSE4213
Prohibitions: None

Synopsis: This subject covers the products, processes, techniques and tools for system validation & verifications including acceptance tests. Commercial Testing Tools from Rational, Mercury Interactive and others will be used to apply in practice knowledge learnt about software testing from a theoritical perspective. Inspection and testing methodologies, analysis of artifacts, robustness, performance analysis configuration management, quality assurance plan and standards including ISO9000/AS39000, compliance, assessment, certification issues are covered. It shows how to predict, analyse and control defects in complex software systems. It introduces verification methods such as inductive methods for safety properties. It covers operational semantics for  sequential and concurrent programs based on Hoare logic - assertion mechanisms - precondition, postcondition and invariants with a view to systematic test planning and validation.

Objectives:

Assessment: Assignments 80%, Demonstration 20%

Meeting Times
Wed. 10am-12 noon (Venue: Bldg.11, H6)


AV Equipment
Supervisor in Menzies Bldg., Ground Floor, Bldg.11, H6  :
 Mr Scott.Doyle,
scott.doyle@its.monash.edu.au,  ph:  53289


Consultation Days
Mondays and Wednesdays (Bldg.75, Room 194)  - 3 - 4pm

Lectures - Week by week Lecture overheads, and required readings - Under Construction and
subject to change; Slides being updated and additional readings will  be assigned)


Announcements: Week 5 Aug 15: Discuss Revised Deadlines; Week 3 (Aug 3): Part of the Lecture session will be devoted to Eclipse, JUnit & Coverage Tool Demonstration. Students are expected to attend the session to get to know these tools in action. Required knowledge for Assignment 1.


Lectures :
 
Week by Week Topic overview

 Prereading, Hurdles and Assignments

Assignment details:

Assignment 1, 40 %, Due Date: Week 7, 31st Aug. 2005

Assignment 2, 
20%, Due Date: Week 9, Sep  14th 2005
Assignment 3,
40%, Due Dates: Week 12,  wed 12th Oct 2005,

Presentations: Week 12 & 13






Lecture
Date
Topic Details
Readings
1
Jul 20
Administration details
Introduction to subject
Assessment Details

Basic Testing  Concepts, IEEE Definition,
 
<29 Slides>
Tool-Validation Compliance - (TVAL-C)  framework   TVAL-C
 

NIST Report - july 2002 (300 + pages)
I V&V

Stephen R. Schach (1996), Testing: Principles and Practice, ACM Computing Surveys, 28:1, 277--279 (1)
SWEBOK KA Description of Software Testing V.1.0 - A. Bertolino, CNR, Italy
shown in terms of SWEBOK Software Testing KAs

Also refer to
http://www.csse.monash.edu.au/~doit  for a mapping of our BSE against SWEBOK.

2
Jul 27
Testing Foundation
Faults vs. Failures 
<30 Slides>
M Young, R N Taylor (1989), Rethinking the taxonomy of fault detection techniques, Proceedings of the 11th International conference on Software engineering
Rethinking the taxonomy of fault detection techniques (local copy for class discussion)
3





4

Aug 3




Aug
10


Software Test Automation
Automated testing: Introduction
<6 Slides>
JUnit testing
<34 Slides>

Automating JUnit, Run Tests from Ant
<17 Slides>
More on Automated testing
<33 Slides>
< Requisite Pro - 9 Slides>
<TVAL-C-4 Slides>
<Semester Start Procedures>
Kai Hsiung Chang et al (1991), Heuristic Approach to Test case Generation , ACM Publ., pp. 174-180
M Fewster & D Graham (1999), Software Test Automation, Addison-Wesley
E Dustin, J Raksha, J Paul (1999), Software Test Automation, Addision-Wesley
Automated Testing tools -
Rational Tools from IBM, Mercury Interactive testing tools, 
Compuware Testing Tools
Eclipse / JUnit / djUnit  demonstration in the MUSE Lab

4
Aug 10
Testing Techniques  <17 slides>
 Testing in Practice <Slides>

 Documentation for Tests <12 slides>
Test case Templates from IEEE  Std  (for use in Class and not for commercial use ) and adapated versions on the web ieeestd-829-1998.pdf
GenericTestPlan.doc
testcase-template.doc
 E. J. Weyuker (1988), The evaluation of program-based software test data adequacy criteria, Communications of the ACM, Vol 31(6)
The evaluation of program-based software test data adequacy criteria (local copy provided for class discussion)

Hong Zhu, Patrick A. V. Hall, John H. R. May (1997),
Software unit test coverage and adequacy, Vol 29(4),  ACM Computing Surveys, pp.366-427
:
Software unit test coverage and adequacy (local copy provided for class discussion)
5
Aug 17
Testing Techniques
Data flow coverage criteria < 19 Slides>
Testing & Certification<10 Slides>
Testing within SQA <12 Slides>
Testing  program components <15 slides>
Test Adequacy and Coverage

Hutchins M, Foster H, Goradia T,  Ostrand T (1994),
Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 16th International Conference Software Engineering (ICSE), 
Page(s): 191 –200

 
Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria (local copy for class discussion)


6
Aug 24
Test case Generation
<25+ slides>

Test Oracles
<31 slides>

Automated GUI Testing
<20 slides>


Kuo-Chung Tai (1993), Predicate-based test generation for computer programs, 15th International Conference Software Engineering (ICSE), pp. 267--276.
Predicate-based test generation (local copy for class discussion)
Effectively prioritizing tests in development environment, ACM publ., pp. 97-106. (local copy for class discussion)

<Assignment 2 - available online)
 


7
Aug 31
Test Maturity Models (TMM)
<http://www.improveqs.nl/uktmmpa.htm >
< http://www.improveqs.nl/>

Software Standards
 
<12 slides>

Mutation Testing
<53 Slides>

Refer to papers in the relevant sections in the pdf file.
http://www.csse.monash.edu.au/~sitar/CSE4431-MUSE2002/V-V-Qual-Stds.pdf
8
Sep 7
Component Testing
Automated Software Engineering ASE 03 Tutorial by Prof. Gao - <29 slides>
Standard for Software Component Testing - BCS SIGIST 2001<67 page pdf>

Model Driven Architectures, Model Transformation
Dagstuhl seminar by Jean Marc Jezequel on Model Transformation
Model Transformation with a dedicated impertive language
http://www.csse.monash.edu.au/~sitar/CSE4431-MUSE2002/V-V-Qual-Stds.pdf
9
Sep 14
Interviews - Assignment 1 http://www.csse.monash.edu.au/~sitar/CSE4431-MUSE2002/V-V-Qual-Stds.pdf
10
Sep 21
Interviews - Assignment 2 http://www.csse.monash.edu.au/~sitar/CSE4431-MUSE2002/V-V-Qual-Stds.pdf


Semester Break Sep 26- Sep 30th 2005

11
Oct 5
Model Driven Development and Testing
<CD - Inside the UML with IBM Rational  from IBM, Sep 2003>
<Towards Model-Driven Testing - http://www.elsevier.nl/locate/entcs/volume82.html>
<Model driven development and testing - A case study - http://modeldrivenarchitecture.esi.es/pdf/paper4-3.pdf>

<Model driven testing in Product family context in ESI>

http://www.csse.monash.edu.au/~sitar/CSE4431-MUSE2002/V-V-Qual-Stds.pdf
12
Oct 12
Presentation - Research papers
http://www.csse.monash.edu.au/~sitar/CSE4431-MUSE2002/V-V-Qual-Stds.pdf
13
Oct 19
Presentation - Research papers http://www.csse.monash.edu.au/~sitar/CSE4431-MUSE2002/V-V-Qual-Stds.pdf


Some URLS of interest:

http://www.sqa-test.com/articles.html 

http://polaris.umuc.edu/~skerby/help/wbib_swe.htm

http://www.irisa.fr/pampa/themes/themes-e.html

http://www.rbsc.com/pages/who_who.html

http://web.comlab.ox.ac.uk/oucl/research/grants/eh.html

http://www.ercim.org/publication/Ercim_News/enw32/lemetayer.html

http://www.irisa.fr/vertecs/welcome-english.html

http://www.newcastle.research.ec.org/pdcs/book/pubs.html

http://www.research.ibm.com/journal/sj41-1.html 

http://www.inf.ethz.ch/personal/meyer/publications/computer/ariane.pdf

http://www.ivv.nasa.gov/

july 2002  report http://www.nist.gov/public_affairs/releases/n02-07.htm

http://www.nist.gov/director/prog-ofc/report02-3.pdf  300page + document

- How to read an Engineering Research Paper from Bill Griswold,
Dept Comp.Sc & Engineering, University California, San Diego,  Standard one page form that he gets the students to use - http://www.cs.ucsd.edu/users/wgg/CSE210/paperform.pdf

Articles for Discussion Topics and Papers of interest in CSE4431(Updated regularly)

Refer to http://portal.acm.org  or google or Hargrave Library for papers such as
Mary Jean Harrold  (2000), Testing: a road map, Proceedings of the Conference on The Future of Software Engineering May 2000, pp. 61-82

Communications of the ACM -   Location: Hargrave-Andrew Library Serials Call Number: 510.7806 A849C,
Location: Caulfield Serials Collection Call Number: 001.6405 ASS

And others listed in the pre-reading section in the table given above and in the “Selected Publications List” available at
http://www.csse.monash.edu.au/~sitar/CSE4431-MUSE2002/ V-V-Qual-Stds.pdf

From www.swebok.org - pdf file - Bourque P, Dupuis R and Moore J W, The Emerging Consensus on the Software Engineering Body of Knowledge,
SEPG 2002, feb 2002, pp.49-56 on Software Testing 

ACM Computing Surveys - Title: ACM computing surveys. [electronic resource]. Continues: Computing surveys 0010-4892 Publisher: New York, N.Y. : Association for Computing Machinery, c1971- Material type: [electronic resource]. Linked resources: Full text available online via ProQuest (restricted access)

OOPSLA

ECOOP

IEEE Transactions on Software Engineering

International Conference on Software Engineering  and …

Some Conferences happening in Software Quality and Testing  in 2002

http://www.soft.com/QualWeek/QW2002/framemaker.pht (Quality Week Europe March 2002)

http://www.qaiusa.com/conferences/oct2002/index.html style="font-size: 9pt; color: black;">22nd Annual International Software Testing Conference
October 21 - 25, 2002 ~ Orlando, FL)
http://www.softdim.com/psqt2002north/

( International Conference on Practical Software Quality Techniques (PSQT)
& International Conference on Practical Software Testing Techniques (PSTT)
PSQT/PSTT 2002 North)

Student Paper

Refer to http://www.csse.monash.edu.au/~sitar/CSE4431-MUSE2002/V-V-Qual-Stds.pdf  for topic areas for your Assignment 3. Choose a topic and ask the lecturer to okay it, as students are expected to choose from a variety of topic areas from the pdf file given or a topic of special interest to the student in the area of software testing.

You may want to look for topics for your paper and links to active researchers  by looking at some of the premier conference URLS such as for Automated Software Engineering (ASE), International Symposium on Software Testing and Analysis(ISSTA), International Symposium Software Reliability Engineering (ISSRE), International Conference Software Engineering (ICSE) and associated/colocated workshops in these conferences and symposiums, which are often excellent as well), and other conferences and Journals in this area. You may find the following URLS useful:

http://www.ase-conference.org/ASE2004/   19th IEEE International conference, Linz, Austria, September 20-25, 2004. Some of the workshops in conjunction with ASE2004 are:

http://fates.cs.auc.dk/  4th International workshop on formal approaches to testing software with ASE04

http://eecs.oregonstate.edu/issta2004/   Boston, Massachusetts, USA, July 11-14, 2004

Some of the workshops in conjunction with International Symposium on Software Testing and Analysis (ISSTA 2004):

 http://www.cs.ucsb.edu/~bultan/tav-web/  Workshop on Testing, Analysis and Verification of Web Services (TAV-WEB 2004)  with ISSTA 2004

http://www.sce.carleton.ca/squall/WERST2004/   Workshop on Empirical Research in Software Testing (WERST'2004)   with ISSTA2004

http://www.issre.org/2004/ 15th IEEE International Symposium Software Reliability Engineering, Saint-Malo, Bretagne, France, Nov 2-5, 2004,  Theme: Achieving software dependability through model driven engineering

http://conferences.iee.org/icse2004/  26th International conference Software Engineering, May 23-38, Scotland, UK (also refer to interesting colocated workshops)

http://www.sce.carleton.ca/squall/  - The Software Engineering Quality Engineering Laboratory, Carleton University, Ottawa, Canada  for some interesting papers

http://www.irisa.fr/triskell/welcome.htm   University of Rennes, France

Student Paper

Must include an abstract, overview of the paper, motivation, literature review, students' contribution,
related work, weakness of the techniques discussed, further work and a summary/conclusion.

Judging for Best paper and possible submission to a conference.

Your paper will be judged for its content , written style & format and sticking to deadline schedules given for submissions.

Content and Style

1. Must include an abstract, overview of the paper, motivation, literature review, students' contribution, related work, weakness of the techniques discussed, further work and a summary/conclusion

2. Use academic style writing and do not use sales or marketing or flowery language

3. Check for typos, grammar,style

4. Use page numbers in the paper submitted

5. References must include refereed articles from Proceedings and Journals and not just from popular magazines or websites. A minimum of six references must be included.

6. Books are fine but cannot be the sole references. Citing must be done properly. Refer to standards for citation in  past Honours theses and/or in the refereed articles you have refereed to.

Page Limit

Paper limit of 10 pages applies for student papers. Use Latex template given  and produce a ps file.

Convert ps to pdf from Linux using ps2pdf  filename command and submit the paper  as a pdf file.

Presentation

1. Refer to Guidelines given for Presentation by Sita for your CSE4002 Presentation#1 seminar in semester 1.

2. Stick to time limit given.

3. Practice talk in front of a mirror at home or with a friend and do not come unprepared/unrehearsed to your presentation.

4. Attendance to all your peers' presentation is compulsory. Marks will be deducted for non attendance.


5.   Participation by asking relevant questions at the end of your peer's presentation carries marks for you asking the question and to the presenter if he/she answers them clearly.  Collusion between presenter and the peer member re: Q/A carries a heavy penalty.

6.   Follow a strict decorum. No whistling or heckling or inappropriate behaviour will NOT be tolerated.

7.   Presentations are to be treated as strictly academic/professional activities.