Automated Builds

How Do You Rebuild A Program?

• Most programming languages require you to compile programs before running them
  • E.g., turn C source files into machine-specific object files
    • Which usually have a .o extension
• It's easy enough to create overlap.o from overlap.c: just run cc overlap.c
• But what if you have many source files?
  • cc *.c is slow and inefficient if only one of your 117 source files has changed
• Worse, what if some files rely on others?
  • Suppose the code in overlap.c uses code in rectangle.c
  • If you change both, but only recompile overlap.c, you'll be using out-of-date rectangles
• Comment on this slide

Automate, Automate, Automate

• Remember rule #2: anything worth repeating is worth automating
  • Computers are good at details, and don't get bored, so have them do repetitive tasks
  • We need:
    • A way to describe the tasks (what things to do)
    • A way to specify the dependencies between them (when to do things)
• Most widely used tool for this is called Make
  • Invented in 1975 by Stuart Feldman at Bell Labs [Feldman 1979]
• The good news is that Make will:
  • Figure out what has changed
  • Work out what is affected by those changes
  • Execute commands to bring things up to date (e.g., by recompiling)
    • Commands are handed to the shell for execution, so they're the same ones you would run yourself
• The bad news is Make's syntax
  • Initially very simple, but has grown more complex over time
  • Has turned into a little programming language in its own right (rule #11)
  • We will ignore the complex bits for now, and take a look at a better way to solve these problems in Backward, Forward, and Sideways
• Comment on this slide

Our Example

• Make doesn't care what language you use
• In fact, it doesn't even care if you're using it to recompile things
  • Often used to run tests, keep web sites up to date, etc.
  • Any task where one action depends on others is a good candidate
• Running example: Sakura Synthesis is studying organic fullerene production
  • Automated laboratory equipment runs experiments in batches to create files like this:

  • Time: 1.2271
    Concentration: 0.0050
    Yield: 11.41
    
    Time: 2.5094
    Concentration: 0.0055
    Yield: 11.20
    
    Time: 3.7440
    Concentration: 0.0060
    Yield: 10.90
    

  • Each experiment produces 20-30 files
    • May eventually have several thousand of them
  • Want to:
    • Generate summary files showing the results for particular trials in tabular form
    • Update a file showing the correlation between concentrations and yields based on those tabular summaries
• Comment on this slide

Hello, Make

• Suppose we already have a program called dat2csv.py that reformats data as a table
  • Use comma-separated values (CSV) for the table
• Put the following into a file called hello.mk:

  • hydroxyl_422.csv : hydroxyl_422.dat
    	python dat2csv.py hydroxyl_422.dat > hydroxyl_422.csv
    

  • Note: that indentation must be a tab character, not eight spaces, or a mix of spaces and tabs
    • Yeah, it's a wart, but we're stuck with it
• Some terminology:

  • Figure 14.1: Structure of a Make Rule

  • hydroxyl_422.csv is the target of the rule
  • hydroxyl_422.dat is its prerequisite
    • We'll see in a moment that a target may have many prerequisites
  • The compilation command is the rule's action
    • A rule may also have many actions
• Run make -f hello.mk
  • Make sees that the CSV file depends on the data file
  • Since the CSV file doesn't exist, Make executes the rule's action (i.e., runs dat2csv.py)
• Run make -f hello.mk again
  • hydroxyl_422.csv is newer than its prerequisite, so the action is not executed
• Comment on this slide

Multiple Targets

• Very few programs consist of a single file
  • This Makefile describes two files that are independent of one another

  • hydroxyl_422.csv : hydroxyl_422.dat
    	python dat2csv.py hydroxyl_422.dat > hydroxyl_422.csv
    
    methyl_422.csv : methyl_422.dat
    	python dat2csv.py methyl_422.dat > methyl_422.csv
    

• When you run make -f double.mk, only hydroxyl_422.csv is compiled
  • The first target in the Makefile is the default target
  • Unless you tell it otherwise, that's all Make will update
• Run make -f double.mk methyl_422.csv to build methyl_422.csv
• Comment on this slide

Phony Targets

• Obviously don't want to have to run Make separately for each possible target
  • That would hardly count as “automation”
• Solution: create a phony target that:
  • Depends on all the things you want to recompile, but
  • Doesn't correspond to any files, and so is never up to date
    • Making it therefore always executes its actions

  • all : hydroxyl_422.csv methyl_422.csv
    
    hydroxyl_422.csv : hydroxyl_422.dat
    	python dat2csv.py hydroxyl_422.dat > hydroxyl_422.csv
    
    methyl_422.csv : methyl_422.dat
    	python dat2csv.py methyl_422.dat > methyl_422.csv
    

  • Now, make -f phony.mk all recreates both .csv files
• Typical phony targets in a software project:
  • "all": recompile everything
  • "clean": delete all temporary files, and everything produced by compilation
  • "install": copy files to system directories
    • Many open source projects are installed by typing make, and then make install
• Comment on this slide

Automatic Variables

• Make defines automatic variables to represent parts of rules
  • These variables are automatically redefined for each rule
  • Names are unfortunately very cryptic
    • Keep a reference card handy

"$@"	The rule's target
"$<"	The rule's first prerequisite
"$?"	All of the rule's out-of-date prerequisites
"$^"	All prerequisites

Table 14.1: Automatic Variables in Make

• Example

  • all : hydroxyl_422.csv methyl_422.csv
    
    hydroxyl_422.csv : hydroxyl_422.dat
    	python dat2csv.py $< > $@
    
    methyl_422.csv : methyl_422.dat
    	python dat2csv.py $< > $@
    

  • Note: by default, Make echoes actions before executing them
  • Putting "@" at the start of the action line prevents this
• Comment on this slide

Pattern Rules

• Most files of similar type (Java, C++, etc.) are compiled the same way
  • Rule #2 again: write a pattern rule specifying what to do in the general case
• Use the wildcard "%" to represent the stem of the file's name in the target and prerequisites
  • Must use automatic variables in the actions
    • This is why they were invented

  • all : hydroxyl_422.csv methyl_422.csv
    
    %.csv : %.dat
    	python dat2csv.py $< > $@
    
    clean :
    	@rm -f *.csv
    

• Notice that there's another phony target in this Makefile
  • make -f rule.mk clean will get rid of all the generated files
  • The -f flag stands for “force”
    • Tells rm not to report an error when trying to delete files that don't exist
• Comment on this slide

Dependencies

• Next step: create an overall summary for each treatment
  • E.g., combine hydroxyl_422.csv and hydroxyl_480.csv to create hydroxyl_all.csv, and similarly for the methyl files, using summarize.py
• hydroxyl_all must be rewritten if it is older than either hydroxyl_422.csv or hydroxyl_480.csv
  • Which must in turn be rewritten if they're newer than their corresponding .dat files
• In this case, also make the results depend on summarize.py and dat2csv.py
  • If the programs change, there's a good chance the data file format has changed
    • Or a bug has been found and fixed
• Updated Makefile, and its output when no CSV files exist, are:

  • all : hydroxyl_all.csv methyl_all.csv
    
    %_all.csv : %_422.csv %_480.csv summarize.py
    	python summarize.py $^ > $@
    
    %.csv : %.dat dat2csv.py
    	python dat2csv.py $< > $@
    
    clean :
    	@rm -f *.csv
    

    python dat2csv.py hydroxyl_422.dat > hydroxyl_422.csv
    python dat2csv.py hydroxyl_480.dat > hydroxyl_480.csv
    python summarize.py hydroxyl_422.csv hydroxyl_480.csv > hydroxyl_all.csv
    python dat2csv.py methyl_422.dat > methyl_422.csv
    python dat2csv.py methyl_480.dat > methyl_480.csv
    python summarize.py methyl_422.csv methyl_480.csv > methyl_all.csv
    rm hydroxyl_480.csv methyl_422.csv hydroxyl_422.csv methyl_480.csv
    

  • Notice that Make automatically removes intermediate files at the end
• Often visualize dependencies as a directed graph
  • Dependencies can have dependencies, and so on

  • Figure 14.2: Visualizing Dependencies

  • Mark all the files that are out of date, and follow arrows to find out what will happen
  • Note: Make can execute actions in any order it wants to, as long as it doesn't violate dependency ordering
• Notice that the rule %_all.csv takes precedence over the rule %.csv
  • Make uses the most specific rule available
• Comment on this slide

Defining Macros

• Often want to define values inside a Makefile
  • E.g., the directory programs are to send their output to
  • Or optimization flags for a compiler
• Do not want to set these options over and over again
  • Rule #3: anything repeated in two or more places will eventually be wrong in at least one
• Solution: define variables (usually called macros)
  • E.g., specify the directory where the lab machine puts data files:

  • INPUT_DIR = /lab/gamma2100
    
    all : hydroxyl_all.csv methyl_all.csv
    
    %_all.csv : %_422.csv %_480.csv
    	python summarize.py $^ > $@
    
    %.csv : ${INPUT_DIR}/%.dat
    	python dat2csv.py $< > $@
    
    clean :
    	@rm -f *.csv
    

  • Define by specifying name and value
  • To get value, put a "$" in front of the name, and parentheses or braces around it
    • Without the parentheses, Make will interpret "$XYZ" as the value of the variable "X", followed by the characters "YZ"
    • Yes, it's ugly
• Sometimes useful to pass values into Make when invoking it
  • E.g., change the input directory
• Rather than rewriting the Makefile each time, you can specify name=value pairs on the command line
  • Define a macro with the default value
  • Override it when you want to
• So:
  • make -f macro.mk sets INPUT_DIR to /lab/gamma2100
  • But make INPUT_DIR=/oldcrankymachine -f macro.mk uses /oldcrankymachine
• Make also looks at environment variables
  • You can refer to ${HOME} in a Makefile without having defined it
  • Example:

    • VAL = original
      echo :
      	@echo "VAL is" ${VAL}
      

    • $ make -f env.mk echo
      

    • VAL is original
      

    • $ make -f env.mk VAR=changed echo
      

    • VAL is changed
      

• Comment on this slide

Analysis

• Pro
  • Simple things are simple to do…
  • …and not too difficult to read…
  • …especially compared to the alternatives
• Con
  • The syntax is unpleasant
  • Complex things are difficult to read…
  • …and even more difficult to debug
    • Best you can do is use echo to print things as Make executes
  • Not really very portable
    • Hands commands to the shell for execution
    • But commands use different flags on different operating systems
    • Do you use del or rm to delete files?
• Alternatives
  • Ant
    • primarily for Java
    • Less platform-dependent…
    • …but just as hard to read and debug
  • Integrated development environments
    • Most are proprietary and platform-specific
  • SCons
    • Give users a library to manage dependencies and actions
    • “Makefile” is actually a small program
    • More powerful, and debuggable, but steeper learning curve
• Once builds are automated, the next step is to run them continuously
  • Every time someone checks something into version control, rebuild the software (or site), and re-run tests
  • CruiseControl and Bitten will both handle the details
• Comment on this slide

Exercises

Exercise 14.1:

How can you stop Make from removing intermediate files automatically when it finishes processing?

Exercise 14.2:

Make gets definitions from environment variables, command-line parameters, and explicit definitions in Makefiles. What order does it check these in?