Securing Deployments with eBPF: How GitHub Breaks Circular Dependencies
GitHub relies on its own platform to host its source code and deployments, creating a unique challenge: if github.com goes down, it can't fix itself. This circular dependency problem extends beyond direct outages—deployment scripts might inadvertently depend on GitHub or internal services, causing failures during incidents. In this article, we explore how GitHub uses eBPF (extended Berkeley Packet Filter) to monitor and block unsafe network calls in deployment code, ensuring that circular dependencies don't escalate into major outages. Below, we answer key questions about this innovative approach.
What is the circular dependency issue GitHub faces with its own deployment?
GitHub hosts all its source code on github.com, making it a heavy user of its own platform. This creates a simple circular dependency: to deploy updates to GitHub, you need GitHub to be operational. If the site goes down, you cannot access the code or releases needed to fix it. GitHub mitigates this by maintaining an offline mirror of the codebase for emergency fixes and pre-built assets for rollbacks. However, more subtle circular dependencies also exist. For instance, a deployment script might try to download a binary from GitHub during an outage, or it might call an internal service that itself depends on GitHub. These hidden and transient dependencies can stall or break deployments when they're needed most. To address this, GitHub designed a new host-based deployment system that proactively prevents such calls.
What three types of circular dependencies does GitHub identify?
GitHub categorizes circular dependencies into three types. Direct dependencies occur when a deployment script explicitly contacts GitHub (e.g., downloading a release) during an outage, causing the script to fail immediately. Hidden dependencies are more subtle: a tool already on the machine might check for updates online when invoked, hanging or failing if GitHub is unreachable. Transient dependencies involve the deployment script calling an internal service (like a migrations API), which then tries to fetch a binary from GitHub; the failure cascades back. These types were identified through a hypothetical MySQL outage scenario, where deploying a config change to nodes becomes impossible if any script or tool depends on GitHub functionality that is down. Understanding these categories helped GitHub design a solution that addresses all layers of potential failure.
How does eBPF help monitor and block unwanted network calls during deployments?
eBPF is a kernel technology that allows safe execution of custom programs in response to system events, such as network calls or system calls. GitHub uses eBPF to write small, efficient programs that run inside the Linux kernel on deployment hosts. These programs can monitor every outgoing network request made by deployment scripts and tools. When a request targets a blocked domain or IP (e.g., GitHub’s API or internal services that might be unstable during an outage), the eBPF program can either log it for analysis or block it outright, preventing the circular dependency from causing a failure. Importantly, eBPF runs at near-native speed with minimal overhead, making it practical for real-time enforcement. By selectively applying these filters based on deployment context, GitHub ensures that only safe, offline-capable operations proceed, breaking the chain of dependencies without modifying the scripts themselves.
Can you walk through a typical scenario where eBPF prevents a deployment failure?
Consider a MySQL outage where GitHub cannot serve release data. An operator needs to run a deploy script on the affected MySQL node to roll out a configuration fix. Without eBPF, the script might try to pull the latest open source tool from GitHub—a direct dependency—and fail because GitHub is down. With eBPF, the kernel program intercepts that network call and blocks it immediately, returning an error to the script but allowing the script to handle it gracefully (e.g., by using an already cached binary). Similarly, if the script uses a tool that checks for updates (hidden dependency), eBPF blocks the update check without hanging. The deployment continues using local resources. In a transient dependency case, the script calls an internal service, but that service's attempt to fetch from GitHub is also blocked at the kernel level, preventing the cascade. The deploy completes using only what's available on the host, making the process robust against dependencies on external or internal services that might be unavailable.
What are the practical steps for writing eBPF programs to improve deployment safety?
To get started, you need a Linux machine with eBPF support (kernel 4.18+). Use tools like bcc (BPF Compiler Collection) or bpftrace to write simple hooks. For GitHub’s use case, they likely create eBPF programs that attach to network syscalls like connect() or sendto(). The program checks the destination IP or domain against a list of known dependency endpoints (e.g., GitHub’s API, internal services). If matched, it can return an error code (e.g., -EPERM) to the caller. More advanced approaches involve filtering by process ID or cgroup to target only deployment-related scripts. You can test your program in a sandbox using bpftrace scripts that print blocked calls. Once verified, load it into the kernel using the bpftool or via a daemon. GitHub likely wraps this in their deployment framework to dynamically enable/disable filters based on the deployment phase. Remember to allow exceptions for genuine connectivity needs during normal operations.
How does this approach compare to traditional methods of dependency checking?
Traditionally, the responsibility fell on each team to manually audit their deployment scripts and identify circular dependencies. This was error-prone and inconsistent across teams. Review tools or pre-deployment checks could catch some issues, but they often missed hidden or transient dependencies that only surface during outages. eBPF shifts the enforcement from code review to runtime, providing a guarantee that no unintended network calls can occur, regardless of what the script does. It also doesn't require modifying the scripts themselves, which is crucial for third-party tools or legacy code. Compared to firewalls or DNS blackholes, eBPF offers fine-grained, process-level control that can be toggled per deployment without affecting other workloads on the same host. This makes the entire deployment system more resilient by design, reducing the cognitive load on teams and ensuring consistent safety across all hosts.
What lessons did GitHub learn from implementing eBPF for deployment safety?
GitHub learned that circular dependencies are more pervasive than initially assumed, especially hidden and transient ones. Early attempts relying solely on script auditing proved insufficient. eBPF provided a low-overhead, kernel-level safety net that could enforce policies without re-architecting existing deployments. A key lesson was the importance of whitelisting legitimate network calls—over-blocking could break expected behavior. They also discovered that eBPF programs must be carefully designed to avoid kernel bugs (e.g., ensure proper context checking). Another takeaway is that logging blocked calls is essential for debugging and updating the filter rules. Overall, GitHub recommends a phased rollout: start with monitoring, then enable blocking for critical dependency paths, and expand over time. This approach not only improved deployment safety but also gave teams confidence to handle incidents more effectively. The use of eBPF is now a core part of GitHub’s deployment infrastructure, and they encourage others to explore it for similar safety challenges.