[tisem, airbnb, template, workflow, example]
Contribute
Tour Tutorials Building Blocks Examples About
Contribute
Building Blocks
Configure your computer
Statistics and ComputationGet Text Editors Install Python Packages Set up Git and GitHub Set up LaTeX Set up MATLAB Set up Pandoc Set up Perl Set up Python Set up R and RStudio Set up Stata
Automation and WorkflowsMaking Path Settings Permanent Set up Command Line Tools Set up Make Set up Snakemake Set up TeamViewer Set up the AWS Client
Research InfrastructureLearn about the various ways to setup a research infrastructure Getting started with Research Cloud Set up SURF Research Drive and rclone to store data
Task-specific ConfigurationsConfiguring Python for Web Scraping
Develop your research skills
Learn to CodeLearn Bash Commands Learn Python and Julia Learn R Learn Regular Expressions
TipsSearching for literature on the web Build and maintain a reference list Cite efficiently in your academic paper Principles for Good Coding Python coding style guidelines R coding style guidelines Stata coding style guidelines
Collect your data
Web Scraping & APIsAvoid Getting Blocked While Scraping Extract Data From APIs Read & Write Data From APIs Scrape Dynamic Websites Scrape Static Websites
Workflows for Online Data CollectionAssess what Data is Available Calculate Sample Sizes for Web Scrapers Monitor and Safeguard Your Data Quality Online Data Collection and Management Safeguard Legal Compliance When Scraping
Store and document your data
Store DataChoose a Data Repository for Long-Term Storage Configure Environment Variables Download Data Programmatically Use AWS S3 Buckets
Document DataDocument New Data README Best Practices
Prepare your data for analysis
Data PreparationImport Large Datasets Into R Manipulate Data Clean Data Scraped From Social Media Data Preparation and Workflow Management
Analyze data
RegressionsRun a Regression Analysis
Machine LearningIn-depth Introduction to Machine Learning and R
Automate and execute your work
Automate your WorkflowUse Makefiles to Re-Run your Code Use this checklist to improve your project's structure Automatically Install R Packages Used in a Project Integrating Stata in Automated Workflows Schedule recurring tasks (e.g., by day, hour) Share Large Temporary Files Use Software Environments to Ensure Replicability
Collaborate and share your work
Version and Share CodeGet Started with Using Git and GitHub The Most Important Git Commands You Should Know Contribute to Open Source Projects Exclude Files From Versioning Make Your Code Citable with DOI Remove Sensitive or Large Files From Your Repository Working with Large Files on GitHub
Write your PaperWrite a Professional Research Paper with LyX Create Slides in No Time with LyX Get Started With Our Bookdown Thesis Template Get Started With Our LaTeX Templates Export Your Tables for Print-ready Publications Cheatsheet for LaTeX Math Commands
Professionalize Project ManagementManage your Projects using the GitHub Project Board How to Prepare for (and Follow-up On) Scrum Meetings Write GitHub Issues That Get the Job Done Bring Open Science Practices to your Research Group
Publish on the WebBuild Interactive Dashboards With R Shiny


A Reproducible Research Workflow with AirBnB Data

Overview

Using publicly available data from AirBnB (available via Kaggle.com), we illustrate how a reproducible workflow may look like in practice.


We’ve crafted this project to run:

  • platform-independent (Mac, Linux, Windows)
  • across a diverse set of software programs (Stata, Python, R)
  • producing an entire (mock) paper, including modules that
    • download data from Kaggle,
    • prepare data for analysis,
    • run a simple analysis,
    • produce a paper with output tables and figures.

How to run it

Dependencies

  • Install Python.
    • Anaconda is recommended. Download Anaconda.
    • check availability: type anaconda --version in the command line.
  • Install Kaggle package.
    • Kaggle API instruction for installation and setup.
  • Install Automation tools.
    • GNU make: already installed in Mac and Linux OS. Download Make for Windows OS and install.
    • Windows OS users only: make Make available via the command line.
      • Right Click on Computer
      • Go to Property, and click Advanced System Settings
      • Choose Environment Variables, and choose Path under the system variables, click edit
      • Add the bin of Make
    • check availability: type make --version in the command line.
  • Install Stata.
    • making Stata available via the command line. Instruction for adding Stata to path.
    • check availability: type $STATA_BIN --version in the command line.
  • Install Perl.
    • Perl is already installed in Mac and Linux OS. Download Perl for Windows OS.
    • Make sure Perl available via the command line.
    • check availability: type perl -v in the command line.
  • Install LyX.
    • LyX is an open source document processor based on the LaTeX. Download LyX.
    • make sure LyX available via the command line.
    • check availability: type $LYX_BIN in the command line.

Run it

Open your command line tool:

  • Check whether your present working directory is airbnb-workflow by typing pwd in terminal
    • if not, type cd yourpath/airbnb-workflow to change your directory to airbnb-workflow
  • Type make in the command line.

Directory structure

Make sure makefile is put in the present working directory. The directory structure for the Airbnb project is shown below.

├── data
├── gen
│   ├── analysis
│   │   ├── input
│   │   ├── output
│   │   │   ├── figure
│   │   │   ├── log
│   │   │   └── table
│   │   └── temp
│   ├── data_preparation
│   │   ├── audit
│   │   │   ├── figure
│   │   │   ├── log
│   │   │   └── table
│   │   ├── input
│   │   ├── output
│   │   │   ├── figure
│   │   │   ├── log
│   │   │   └── table
│   │   └── temp
│   └── paper
│       ├── input
│       ├── output
│       └── temp
└── src
    ├── analysis
    ├── data_preparation
    └── paper
  • gen: all generated files such as tables, figures, logs.
    • Three parts: data_preparation, analysis, and paper.
    • audit: put the resulting log/tables/figures of audit program. It has three sub-folders: figure, log, and table.
    • temp : put the temporary files, such as some intermediate datasets. We may delete these filed in the end.
    • output: put results, including the generated figures in sub-folder figure, log files in sub-folder log, and tables in sub-folder table.
    • input: put all temporary input files
  • data: all raw data.
  • src: all source codes.
    • Three parts: data_preparation, analysis, and paper (including TeX files).
Edit this page

Related
Posts