Automating Workflows
Up to this point, you should have created the following five R scripts:
Summary| File | Description | Phase |
|---|---|---|
download.R |
Downloads the data from Inside Airbnb and stores as csv format |
Input |
clean.R |
Preprocesses the raw data into an aggregated format ready for analysis and visualisation |
Transformation |
pivot_table.R |
Create a pivot table for the number of reviews by region across time |
Transformation |
plot_all.R |
Create a line chart for the total number of reviews in a city across time |
Output |
plot_Amsterdam.R |
Create a line chart for the number of reviews for the top 3 neighborhoods in Amsterdam |
Output |
As you’ve worked through the set of exercises, you’ve repeatedly built on preliminary results. For example, the plot for the top 3 cities in Amsterdam (plot_Amsterdam.pdf) could only be created once the pivot_table.csv file had been generated. In a similar way, the preprocessing pipeline (clean.R) could only take place once the data (listings.csv & reviews.csv) had been downloaded. These dependencies have been depicted in the figure below.
Revisit the study notes on “Automating your Pipeline” and write a makefile that captures the end-to-end process (from download.R to plot_all.pdf & plot_Amsterdam.pdf). Also, add an all and clean phony target.
Exercise
Swap the url_listings and url_reviews for a historical dataset of Amsterdam from the previous year (gather the links from the “show archived page"). Run make again in the root directory.
Do the same for a recent Airbnb dataset from New York. If done correctly, it should not take more than a minute (power to automation!). Do your workflows still run as expected? How about the plot_Amsterdam.R file? Why is that?
