Contribute
Tour Tutorials Building Blocks Examples About
Contribute
Building Blocks
Configure your computer
Statistics and ComputationGet Text Editors Install Python Packages Set up Git and GitHub Set up LaTeX Set up MATLAB Set up Pandoc Set up Perl Set up Python Set up R and RStudio Set up Stata
Automation and WorkflowsMaking Path Settings Permanent Set up Command Line Tools Set up Make Set up Snakemake Set up TeamViewer Set up the AWS Client
Research InfrastructureLearn about the various ways to setup a research infrastructure Getting started with Research Cloud Set up SURF Research Drive and rclone to store data
Task-specific ConfigurationsConfiguring Python for Web Scraping
Develop your research skills
Learn to CodeLearn Bash Commands Learn Python and Julia Learn R Learn Regular Expressions
TipsSearching for literature on the web Build and maintain a reference list Cite efficiently in your academic paper Principles for Good Coding Python coding style guidelines R coding style guidelines Stata coding style guidelines
Collect your data
Web Scraping & APIsAvoid Getting Blocked While Scraping Extract Data From APIs Read & Write Data From APIs Scrape Dynamic Websites Scrape Static Websites
Workflows for Online Data CollectionAssess what Data is Available Calculate Sample Sizes for Web Scrapers Monitor and Safeguard Your Data Quality Online Data Collection and Management Safeguard Legal Compliance When Scraping
Store and document your data
Store DataChoose a Data Repository for Long-Term Storage Configure Environment Variables Download Data Programmatically Use AWS S3 Buckets
Document DataDocument New Data README Best Practices
Prepare your data for analysis
Data PreparationImport Large Datasets Into R Manipulate Data Clean Data Scraped From Social Media Data Preparation and Workflow Management
Analyze data
RegressionsRun a Regression Analysis
Machine LearningIn-depth Introduction to Machine Learning and R
Automate and execute your work
Automate your WorkflowUse Makefiles to Re-Run your Code Use this checklist to improve your project's structure Automatically Install R Packages Used in a Project Integrating Stata in Automated Workflows Schedule recurring tasks (e.g., by day, hour) Share Large Temporary Files Use Software Environments to Ensure Replicability
Collaborate and share your work
Version and Share CodeGet Started with Using Git and GitHub The Most Important Git Commands You Should Know Contribute to Open Source Projects Exclude Files From Versioning Make Your Code Citable with DOI Remove Sensitive or Large Files From Your Repository Working with Large Files on GitHub
Write your PaperWrite a Professional Research Paper with LyX Create Slides in No Time with LyX Get Started With Our Bookdown Thesis Template Get Started With Our LaTeX Templates Export Your Tables for Print-ready Publications Cheatsheet for LaTeX Math Commands
Professionalize Project ManagementManage your Projects using the GitHub Project Board How to Prepare for (and Follow-up On) Scrum Meetings Write GitHub Issues That Get the Job Done Bring Open Science Practices to your Research Group
Publish on the WebBuild Interactive Dashboards With R Shiny


Practicing Pipeline Automation using Make

Pipeline Automation Overview
Verify Software Setup
Download the Template
Running the Workflow
Inspecting the Data-preparation Pipeline
Dry-run of Make and First Modifications
More Modifications
Extending the Analysis
Wrap-Up

Let’s modify the analysis

We’re done with the data-preparation pipeline for now, and turn our attention to the analysis part of our project.

Admittedly, there’s not much here yet. Try fiddling around with the files a bit when you proceed to our practice questions and answers below.

Now let’s continue with a couple of modifications. You can directly start working on the practice questions below.

Practice questions and answers

  1. Recall the powerful makefile we’ve introduced to you a while ago? Well, open the makefile in src/analysis/ now and try to understand the steps of this stage of the pipeline! What happens, exactly?

  2. Let’s open preclean.R (e.g., in RStudio). Try to understand what this script does. Then, filter the data only for tweets with polarity>0.

  3. Last, provide some summary statistics (summary(dt$nwords)) of the word count, and produce a histogram hist(dt$nwords) of it in the RMarkdown document.

Watch the solution here

Edit this page