Automating Pipelines with
Make is a "build tool", allowing us to control the execution of a set of command line statements to assemble pipelines that define what gets executed when. Using
our entire workflow becomes transparent and reproducible.
Why Build Tools?
Imagine you have built a complex project, consisting of dozens of datafiles and scripts. Also imagine you haven't worked on the project for a few weeks, and now wish to continue your work where you left off.
The big questions then become:
- In which order do the files need to be executed?
- Which files are up-to-date and do not have to be executed again?
For this purpose, we use a workflow management system - or, in technical terms - "build tool". Build tools will allow us to control the execution of a set scripts by by running them from the command line.
Some reasons we push this topic are:
- Your workflow / order of execution is explicitly documented.
- Each time you run
make, it only executes each script if the output is expected to be different from the last time your ran it. That is, it runs 'partial builds.'
- If multiple users work on the project, they can easily execute code that others have written.
We will use
make to automate the execution of our projects with a "single click", so that our entire work flow is reproducible.
For Windows Users
We will install
make so that it plays nicely with your Anaconda/Python distribution and the Windows command line.
Watch our YouTube video, in which we walk you through the setup on Windows.
make available via the PATH settings on Windows
We need to update our PATH settings; these settings are a set of directories that Windows uses to "look up" software to startup.
- Open the settings for environment variables
- Right-click on Computer.
- Go to "Properties" and select the tab "Advanced System settings".
- Choose "Environment Variables"
Alternatively, type "environment variable" (Dutch: omgevingsvariabelen) in your Windows 10 search menu, and press Enter.
Pathfrom the list of user variables. Choose
Windows 7 and 8 machines: If you chose your installation directory to be
C:\Program Files\GnuWin32\binduring your installation (i.e., you did use the default directory), copy and paste the following string without spaces at the start or end:
;C:\Program Files (x86)\GnuWin32\bin
Windows 10 machines:
Newand paste the following string:
C:\Program Files (x86)\GnuWin32\bin
- Click on
OKas often as needed.
For Mac Users
- Please install X-code command line tools, which includes
Open a terminal by searching for it with spotlight,
cmd + spacebar then type terminal and press
Return when it appears. Then, copy and paste the following:
For Linux Users
Make is pre-installed on Linux operating systems so there is nothing to be done.
Verifying Your Installation
To check that
Make is installed and working correctly, open a terminal session and type (then hit the return key):
If everything is working correctly, you should get the following output:
make: *** No targets specified or no makefile found. Stop.