Overview

It is useful to break down a project into its most basic parts, which we call pipeline and components.

• A pipeline refers to the steps that are necessary to build a project (e.g., prepare dataset, run model, produce tables and figures), and

• Components refer to a project’s most nuclear building blocks (e.g., data, source code, and generated temporary and/or output files).

Later on, you will see that such a structure enables you to work on your project using multiple computers (e.g., your workstation, your laptop, a computer in the cloud), or with various collaborators/co-authors.

Project Pipelines

In a research project, one typically has several tasks to accomplish, such as preparing the data, analyzing the data, writing the paper and producing a set of slides.

This is what we call a “project pipeline."

A typical pipeline for an academic paper may look like this:

Over time, your pipeline will grow increasingly complex. For example, the pipeline above recently “matured” into this one:

Of course, you can even come up with pipelines that are even more complex - not because we love complexity, but because we love clarity.

The benefits of conceiving your project like a pipeline are straightforward:

• Write clearer source code: separate your thousand-line source code into multiple smaller, more accessible components (as defined in your pipeline, like preparing dataset, then running a model, then producing tables and figures).

• Obtain results faster: Because your project is separated into different pipeline stages and each of these stages is self-contained, you can easily run “later” stages of your project (we call this “downstream”), based on different input files defined earlier in your project (we call this “upstream”). You can run later stages in your project (e.g., “analysis”) using different inputs (e.g., a data set at the monthly level, and at the weekly level) - that way, you can rapidly understand how robust your findings are with regard to alternative specifications.

• Increase transparency and foster collaboration: With more transparent source code, you allow others to more easily understand your code. Also, why not make a co-author responsible for prepping that data for you (“upstream” in your pipeline), while you concentrate on developing a prototype model (more “downstream” in your pipeline)?

• Use multiple software packages: In a pipeline, the “outputs” of one module serve as the “inputs” of another. Hence, you can use multiple software packages to accomplish different tasks in your pipeline. For example, we use R’s powerful data.table package to clean data, and pass on the cleaned dataset to a clustering algorithm in Python’s scikit-learn. In other words, we use whatever software package we deem most appropriate for each step in our pipeline.

Project Components

Now that we’ve covered what a pipeline is, let’s draw our attention to project components, which are the most nuclear building blocks of a project. It’s useful to think of these components as separate entities of your project because their nature allows you to apply different data management policies.

• For example, we probably all agree it’s desirable to roll back to previous versions of a project (e.g., an earlier version of a prepped dataset). But - if you work on large datasets - it may probably be too burdensome to store each version of your generated files (e.g., in one of the projects we’ve been working on, the generated (cleaned) data sets were a 500 GB, and we’ve created probably close to 50 versions = 25 TB).

• If you think about this a bit more, you may discover that storing these different data sets is completely inefficient - as the combination of raw data and versioned source code will be able to “re-cast” any data set version you have ever worked on.

Suppose you needed to bring structure to a chaotic and poorly organized project, then probably you may end up with the following project “components”:

1. raw data

   These are the data sets used in the projects. Typically, you will never delete that data, but only add new data sets throughout the duration of the project.

2. source code

   This is all the source code written to execute your pipeline (defined earlier; e.g., preparing data, analyzing it, and producing some tables or figures).

   • Each source code file typically…

     • loads some input files or accepts some arguments,
     • does some operations on data (e.g., clean it, estimate a model), and
     • then saves generated files (output, audit, or temporary files).

   • It’s important to version your code, so that you can always roll back to different incarnations of your project.

3. generated files (temporary, output, and audit)

   These files are the result of executing your source code. Think about cleaned data sets, results of your analysis, tables and figures, or some log files to audit whether everything went well (e.g., counting the number of observations, and writing those in audit.txt). In most instances, it may be enough to store those only temporarily on your local computer.

4. file exchange

   File exchanges are used to easily “ping-pong” (upload or download) generated temporary or output files between different stages of the pipeline.

5. notes

   You can keep your meeting notes, PDFs of the literature that you’ve read, etc. in a separate folder (with, potentially, subfolders).

Putting it all together…

It’s time to finally build your project! See our overview chart below, which illustrates how different stages of your pipeline use different project components.