Automating Workflows
Up to this point, you should have created the following five R scripts:
Summary
| File | Description | Phase |
|---|---|---|
download.R | Downloads the data from Inside Airbnb and stores as csv format | Input |
clean.R | Preprocesses the raw data into an aggregated format ready for analysis and visualisation | Transformation |
pivot_table.R | Create a pivot table for the number of reviews by region across time | Transformation |
plot_all.R | Create a line chart for the total number of reviews in a city across time | Output |
plot_Amsterdam.R | Create a line chart for the number of reviews for the top 3 neighborhoods in Amsterdam | Output |
As you've worked through the set of exercises, you've repeatedly built on preliminary results. For example, the plot for the top 3 cities in Amsterdam (plot_Amsterdam.pdf) could only be created once the pivot_table.csv file had been generated. In a similar way, the preprocessing pipeline (clean.R) could only take place once the data (listings.csv & reviews.csv) had been downloaded. These dependencies have been depicted in the figure below.
Revisit the study notes on "Automating your Pipeline" and write a makefile that captures the end-to-end process (from download.R to plot_all.pdf & plot_Amsterdam.pdf). Also, add an all and clean phony target.
Exercise
Swap the url_listings and url_reviews for a historical dataset of Amsterdam from the previous year (gather the links from the "show archived page"). Run make again in the root directory.
Do the same for a recent Airbnb dataset from New York. If done correctly, it should not take more than a minute (power to automation!). Do your workflows still run as expected? How about the plot_Amsterdam.R file? Why is that?