Software Engineering and Programming Fundamentals
"The best programs are written so that computing machines can perform them quickly and so that human beings can understand them clearly. A programmer is ideally an essayist who works with traditional aesthetic and literary forms as well as mathematical concepts, to communicate the way that an algorithm works and to convince a reader that the results will be correct." -- Donald Knuth
Lesson 1: What is software engineering?
Do we have to apply software engineering practices to every
No, we do not?
On a small program you are writing for your own convenience, or just for fun, you can pretty much ignore SE.
When should we apply SE?
As projects increase along three dimensions:
- The number of people involved.
- The amount of code involved.
- The amount of time we expect the program to be in use.
Why? Well, as the number of people increases, we need to make
our code easier to understand, to bring people up to speed
As the amount of code increases, we must apply SE to keep it comprehensible, even for a single coder.
As the expected life of the program increases, we must make the program easier to change.
Modularity: Our Main Tool
The number one friend of the software engineer is modularity. Good software is composed of components, or modules, that are as independent as possible. They interact with each other through narrow interfaces.
Consider the spark plug.
The spark plug is a modular component in your car. When you need new sparks, you just plug them in, and off you go. Even if there is a new, higher grade spark plug available, as long as it plugs into its interface to the rest of the system with no trouble, it will work.
Imagine your surprise if, when you went to replace the sparks
in your car, you were told by the garage that you'd also have
to replace the brakes, the radio, and the windshield wipers!
"Why," you'd ask, "are they faulty?"
"Nope: they just aren't compatible with the new spark plugs."
Badly engineered software is like that: to change one part, you often need to change many others. This makes each change expensive, and likely to introduce bugs, since it is hard to keep track of all the parts a change might affect.
Focus on Interfaces!
Again, let's think about a car. This time, we will look at the gas pedal.
The gas pedal is an outstanding example of a narrow
The driver (the 'user'!) interfaces with the system that
accelerates the car with only two inputs:
* Push harder, go faster!
* Ease up, slow down!
The great thing about narrow interfaces is that the engineers can change almost everything about a component, without the users of the component noticing, so long as they don't change the interface. Say, one night, a team of dedicated eco-activists break into your garage and replace your gasoline engine with an electric one. So long as the "gas" pedal still works the same way, you might not even notice, at least until you tried to refill your gas tank.
Early musical synthesizers were examples of an engineered product that did not have a narrow interface:
By contract, in modern synths, the player typically just presses a few buttons to get a new sound.
One reason we are going to focus on building an API server is that doing so puts our focus squarely on interfaces. We will put our early design efforts into thinking about what interface we should provide to the user of our API server.
Lesson 2: Some programming fundamentals
Humans have been programming computers for seven or so decades now. Let us look at some findings on how to write the best software possible!
This stands for Don't Repeat Yourself! It means that any part of your system that might ever need to change should have a single place where you can make the change. Don't copy blocks of code to wherever you need them in your program: write a function and call it from each of those places. Don't define your data tables in your database, and also in your code: find a way (like the Django
models.pyfile) to define your data one place and use that definition to generate both the database and the code that uses the DB.
No magic constants.
This is a special case of DRY. It is very tempting, when coding your NYU scheduling app, to write code assuming there are two (major) semesters per year. This will be fine... until NYU adopts a tri-mester system. Instead, define a constant
NUM_SEMS = 2. You might get away with writing
day_of_week = day mod 7, since that number probably will never change. But you really ought to write
hour_of_day = hour mod CLOCK_PERIOD, since both 12 and 24 hour timekeeping methods exist.
Make functions do one job.
Funcitons that perform a single job are simpler to understand, easier to change or eliminate, and render the overall system more comprehensible. For instance, if the county writes a tax program with a function called
calc_taxes, it would be natural to eliminate that function if the job is later passed off to a microservice running on the cloud. But, if the coders also happened to include the code to clear tax liens (county claims against the property for unpaid taxes) in the same function... Oops! No one who ever had a tax lien can sell their property, because the lien never gets cleared.
Keep functions short.
This is related to the previous principle, but focuses on the size of the one job that should be done. A function named
handle_yearly_taxes()is doing one job, but probably way to big a job. It would make more sense to have
record_payments(), and perhaps more.
Format and indent properly.
Different languages have different conventions for how to name variables (
MixedCase, and so on), how to space operators, where to put braces, and so on. You should follow those conventions, unless there is a strong reason not to. Consistent indentation is especially important: it allows a reader of your code to easily line up blocks of control. Irregular indentation is a significant source of bugs, as people modifying the code will make mistakes, for example, about which
elsegoes with which
Code should contain some comments, especially things like docstrings for classes that can be extracted to produce a guide to the system, and comments explaining what particularly tricky or unusual bits of code do. But commenting is no substitute for writing clear, readable code in the first place! The best explanation of what your code does is, if you write it correctly, your code itself. Remember that we could, and once did, write code just as a sequence of 1s and 0s. And all higher-level languages need to be translated into such code in the end. So why bother with C, Java, or Python? These languages exist for humans, not for computers: they make it easier for us to understand and reason about what a program will do. The upshot: you should look at your code as being every bit as much about communicating to humans as about directing a computer.
Go for the golden mean in
Sometimes, names of functions and variables can be way too cryptic: there are examples in the widely used CLRS Algorithm book where I have found as many as six single-letter variable names used at once. On the other hand, naming a function something like
take_input_of_employee_w2_and_calculate_employee_tax_rate()is absurdly long: please remember, other programmers will have to type your function names in order to call your functions! Such immense names also make it extremely difficult to stay within guidelines like PEP 8's dictum of "no lines longer than 79 characters." A more reasonable middle ground might be something like
calc_tax_rate(), where an employee's W2 might be a parameter for the function.
- Break your code up into modules that handle one aspect of the program... your accounting program might have modules for taxes, payroll, invoices, bills, and bank accounts (perhaps).
- Keep interfaces between modules narrow (as little data has to pass between them as possible) and clearly defined. Change these interfaces as little as possible.
Test, test, test!
Test small pieces of code as you go along. Write an automated test to go with every program or new feature you write. Test as completely by hand as you can: don't just test that your code fetches the data from the DB correctly: test that it still works properly if there is no data in the DB, or, indeed, if there is no DB! ("Properly" here could mean "Display an informative error message instead of crashing.")
Lesson 3: Python coding standards
For this lesson, please read the Python coding standard, PEP 8. It is a very good example of what a coding standard is like, and most of the guidelines can be applied in other languages.