Automated Testing

Lesson 1: The Whys and Hows of Automated Testing

This section is co-authored by Denis Petelin and Prof. Callahan.

  1. Why do we test? We test to see if the software does what we want it to do.
    And whenever it doesn't, what are the effects?
  2. Classic way to do testing: test cases, bugs, regression, growing "regression debt".
    Please study regression testing .
  3. Revolutionary idea: zero-bug mindset: bugs are not tasks in backlog, you have to fix them as you go -- by the end of the day zero bugs exist.
    This is part of the "incremental improvement" idea: if we do continuous integration and continuous delivery with small batches, each delivery into production should produce at most a "small batch" of bugs! And we can fix that small batch immediately.
  4. Revolutionary idea: zero-length feedback, developers can test and fix bugs immediately.
    Part of this is establishing a culture of trust: If we need to feed approval for each change through multiple levels of bureaucracy, we can't fix bugs right away. And we can relate this back to the discussion in chapter one on "Taylorism" versus "Toyota Production System": in the former, the "workers" just carry out the plans of the "managers." In the latter, everyone is responsible for the entire production process.
  5. Test pyramid:
    1. Unit tests for individual classes and methods (models, controllers, views)
    2. Integration tests to check feature top-down.
    3. Acceptance test to check feature as user sees it.
  6. Terminology:
    1. TestCase: set of checks to be performed.
    2. Fixture: prepared data to be loaded into the db.
    3. Fake: a real object created for the test.
    4. Stub: a crude imitation of real object returning hard-coded values.
    5. Mock: an elegant imitation of the object (if real object is not yet ready or expensive).
    6. Test suite: set of tests serving specific purpose. Always: smoke test, main success [AKA happy path], extended tests.
  7. Django benefits:
    1. No need to unit-test Models (except custom query sets & business logic methods).
    2. No need to integration-test Autogenerated View (except live tests).
  8. Anatomy of test case:
    1. setUp()
    2. Test_whatYouTest_whatYouDo_whatYouExpect.
    3. Arrange -- Act -- Assert.
    4. Assert kinds.
    5. tearDown()
  9. Preparing data -- AutoFixture
    1. TaskModelTransactionTestCase(TransactionTestCase): regular fixture.
    2. For lazy guys -- AutoFixture :)
    3. fixture.create()
  10. Typical mistakes:
    1. Useless tests -- testing default Models methods, for example.
    2. Testing implementation -- method save_changes() returns OK -- everything is OK! (Test should check if changes indeed persisted).
    3. Large tests? Fat controllers! 
    4. Refactoring:
      1. Small methods -- less than a screen.
      2. Small tests -- 8-10 lines.
      3. Refactoring palette in the PyCharm.
  11. Good beginners pattern:
    1. Create Model. Add tests if there are custom methods.
    2. Create Controller (View as Django calls it). Test if does what it should do. Test if it handles errors.
    3. Write View (Template as Django calls it). Write LiveTestCase using requirements.
    4. Why preparing requirements still matters (“Please show balance" in Danfoss).
  12. Big idea: can we somehow make requirements document testable?
    1. Turning use cases into tests -- gherkin
    2. Feature file & steps
    3. Passing info around -- context
    4. Selenium -- driving real browser around
    5. Behave test runner (behave-Jango)
    6. JIRA: acceptance tests are now part of the 
    7. Relying strictly on this type of testing is bad idea! (See execution time for one test vs whole suite!)
Lesson 2: Testing Frameworks
Python testing with pytest! Part 1: Introductions and motivating testing.
Python testing with pytest! Part 2

pytest documentation

Lesson 3: Jenkins

Jenkins is a CI/CD tool, but it is used to force the running of automated tests as part of the build process, so we have included it in this section.

Introduction to Jenkins by Denis Petelin

Jenkins on Wikipedia.

Other Material
What Is Jenkins?
Quiz

    'zero-length feedback' means...?

    1. developers can test and fix bugs immediately
    2. the length of a CI/CD pipeline should be 0
    3. all tests should take 0 time
    4. all of the above

    One common way to trigger a Jenkins build is....?

    1. type some new code in an IDE
    2. run make
    3. run Docker
    4. push a commit in a version control system

    Unit tests work primarily at the level of...?

    1. individual classes and methods
    2. user-level interactions
    3. checking the integration of the various modules of the software
    4. all of the above

    Acceptance testing tests features as...?

    1. the computer sees them
    2. the tester sees them
    3. the user sees them
    4. the regulator sees them

    The scripting language for Jenkins is called...?

    1. Groovy
    2. Movie
    3. Groupie
    4. Snoopy

    Jenkins uses a __________ architecture?

    1. master-master
    2. master-agent
    3. peer-to-peer
    4. monolithic

    The "zero-bug mindset" means that...?

    1. no developers should be bugging out
    2. no bug backlog should ever build up
    3. no one should ever make a coding mistake
    4. we should ridicule anyone who introduces a bug

    In pytest, assert statements are a way to...?

    1. practice getting what you really want from life
    2. push some code into production
    3. assert what code should be run
    4. state what condition should be true at the end of a unit of test code

    Testing Python code is aided by a package called...?

    1. pylib
    2. numpy
    3. scipy
    4. pytest

    What is regression testing?

    1. a very regressive form of testing
    2. a way to make tests more primitive each go-round
    3. a way to see if the program has regressed after a change
    4. none of the above