Getting Started with Webhooks: Part 1 - Webhook Servers

This is a two-part guide to the fascinating world of webhooks. In the first part of this series, we'll focus on webhook servers, what they are, how they work, and subsequently deploy a simple webhook server using Node.js and Express on Railway. In the second part, we'll focus on webhook clients, what they are, how they interact with webhook servers, and then deploy a simple webhook client using Streamlit on Railway. Different folks use the webhook server and client labels interchangeably; I'm going with the Wikipedia version here.

What is a Webhook?

Our world is more intertwined than ever - a digital experience on any given day is the (mostly) harmonious result of a complex web of interconnected systems. These systems are often built using different languages and frameworks, deployed in different locations and modes, billed using different consumption models, and yet, share a common trait - they need to communicate with each other and share data.

Data is typically shared across heterogenous systems in two ways, via either a push or a pull mechanism. Push notifications often take the form of webhooks. A webhook is an HTTP request (or, in the words of Wikipedia, "user-defined HTTP callbacks"), triggered by an event in a source system and sent to a destination system, often with a payload of data. According to Hookdeck, webhooks provide a way for one system (the source) to "speak" (HTTP request) to another system (the destination) when an event occurs, and share information (request payload) about the event that occurred.

Often used in SaaS platforms like GitHub, Shopify, Stripe, Twilio, and Slack, webhooks are a powerful tool for web developers to receive real-time notifications and data updates from other web applications. Webhooks are versatile and easy to use, and can be used to trigger a wide range of actions, such as sending an email, updating a database, or triggering further downstream actions. To receive webhook requests, you have to register for one or more of the events (also known as topics) for which the platform offers a webhook. Once the webhook registration for an event is complete, you will receive webhook requests at the destination URL you provided each time the event occurs.

The primary alternative to webhooks is polling-based integration, where a client repeatedly requests data from a server at fixed intervals. While this method can be simpler to set up, it can lead to increased latency and resource consumption.

Getting Started with Webhook Servers

A webhook server is the system that receives and processes webhook requests. It listens for incoming HTTP requests at a specific URL and takes appropriate action based on the data provided. Here are the general steps to be followed when setting up a webhook server:

1. Choose a server-side technology: A webhook server can be built using various server-side technologies, such as Node.js, Ruby on Rails, or Django.
2. Create an endpoint: Define a URL that will receive webhook requests. This URL should be unique and hard to guess, as it acts as a security measure.
3. Implement request handling: Create a function that processes incoming requests, extracting relevant data and initiating necessary actions.
4. Deploy your webhook server: Choose a hosting platform and deploy your server for public access.

There are several webhook servers that are easy to set up, depending on your specific needs e.g. RequestBin, Pipedream, Ngrok, and more. In this post, we'll create and deploy a simple webhook server using Node.js and Express. Here is the sample server-side code; you can find the GitHub repository here.

const express = require("express");

// Create an Express app and listen for incoming requests on port 3000
const app = express();
const PORT = 3000;

// Use middleware to parse incoming requests with JSON and URL-encoded payloads
app.use(express.json());
app.use(express.urlencoded());

// Handle GET requests to the root URL
app.get("/", function(req, res) {
  console.log(req.body);
  res.send("Welcome to the Webhook Server!");
});

// Handle POST requests to specific URLs i.e. webhook endpoints
app.post("/webhook-1", function(req, res) {
  console.log(req.body);
  res.send("Webhook 1 successfully received.");
});

app.post("/webhook-2", function(req, res) {
  console.log(req.body);
  res.send("Webhook 2 successfully received.");
});

// Start the server and listen for incoming connections
app.listen(PORT, () => {
  console.log(`Server running at https://localhost:${PORT}/`);
});

Deploy a Webhook Server with Node.js on Railway

Railway is a modern app hosting platform that makes it easy to deploy production-ready apps quickly. Railway offers persistent database services for PostgreSQL, MySQL, MongoDB, and Redis, as well as application services with a GitHub repository as the deployment source. For the latter, Railway can automatically determine the application runtime and deploy the service. Since we are just testing the waters, Railway's free tier should be sufficient to host the service.

Sign up for an account with Railway using GitHub, and click Authorize Railway App when redirected. Review and agree to Railway's Terms of Service and Fair Use Policy if prompted. Launch the Webhooks one-click starter template (or click the button below) to deploy the webhook client and server instantly on Railway.

We're deploying two services here - one each for the webhook client and server. For each, you'll be given an opportunity to change the default repository name and set it private, if you'd like. Accept the defaults and click Deploy; the deployment will kick off immediately.

Once the deployment completes, the webhook server will be available at a default xxx.up.railway.app domain - launch this URL to access the app. If you are interested in setting up a custom domain, I covered it at length in a previous post - see the final section here.

We'll explore webhook clients at length in the next post but, for simply testing the just deployed webhook server, you can use an API client like Postman, Hoppscotch or HTTPie. Send an HTTP POST request to the endpoint /webhook-1 to elicit the response Webhook 1 successfully received. That's it! Of course, please bear in mind this is an extremely simplistic example, and should not be used as-is in any production capacity without appropriate security measures.

In the next part of this series, we'll focus on webhook clients, their interaction with webhook servers, and deploy a simple webhook client using Streamlit on Railway.