Build Log Streaming: How We Stream Real-Time Build Logs to Your Browser
When you deploy on Deployxa, you see the build logs streaming in real time in the dashboard: each line of the build output appears as it is generated, which gives you immediate feedback on the build's progress. Building a real-time log streaming system that handles concurrent builds, delivers logs with sub-second latency, and scales to thousands of users is a significant engineering challenge. Here is how the system works.
The direct answer is that Deployxa's build log streaming has four components: the build process (which generates log lines), the log collector (which captures log lines and publishes them to Redis), the WebSocket server (which subscribes to Redis and streams log lines to connected clients), and the dashboard (which displays the log lines in real time). The system delivers log lines with sub-second latency, which means you see the output as it is generated. For more on real-time systems, see our article on how we built the logging pipeline.
The Four Components
1. The build process
The build process (e.g., npm run build) generates log lines on stdout and stderr. The Deployxa engine captures these log lines (via the container's log driver) and sends them to the log collector.
2. The log collector
The log collector receives log lines from the build process and publishes them to a Redis channel (e.g., build-logs:{deployment_id}). The collector also stores the log lines in a persistent store (for historical queries).
3. The WebSocket server
The WebSocket server maintains WebSocket connections with the dashboard clients. When a client connects (e.g., to view build logs for a specific deployment), the WebSocket server subscribes to the deployment's Redis channel and forwards log lines to the client.
4. The dashboard
The dashboard is a React app that displays the build logs in a terminal-like view. It establishes a WebSocket connection to the WebSocket server, receives log lines, and appends them to the log viewer. The log viewer supports auto-scroll (to follow the latest output), search (to find specific lines), and copy (to copy the full log).
Step-by-Step: How a Build Log Line Flows
Step 1: The build process generates a log line
The build process (e.g., npm run build) generates a log line: > Building....
Step 2: The log collector captures the log line
The Deployxa engine captures the log line (via the container's log driver) and sends it to the log collector.
Step 3: The log collector publishes to Redis
The collector publishes the log line to the Redis channel build-logs:{deployment_id}.
Step 4: The WebSocket server receives the log line
The WebSocket server (which is subscribed to the Redis channel) receives the log line and forwards it to all connected clients that are viewing the deployment's build logs.
Step 5: The dashboard displays the log line
The dashboard receives the log line via the WebSocket and appends it to the log viewer. The log line appears in the dashboard within sub-second latency.
Common Pitfalls and Troubleshooting
The first pitfall is log volume. A build can generate thousands of log lines (especially for large apps), which can overwhelm the WebSocket server. The fix is to batch log lines (send them in groups, not individually) and to use a scalable message broker (like Redis). The second pitfall is WebSocket disconnections. If the WebSocket disconnects (e.g., due to a network issue), the client misses log lines. The fix is to implement reconnection (with a backoff) and to send missed log lines (from the persistent store) on reconnection. The third pitfall is backpressure. If the client is slow (e.g., due to a slow browser), the WebSocket server might queue up log lines, which can cause memory issues. The fix is to implement flow control (e.g., drop old log lines if the client is too slow). The fourth pitfall is not storing logs. If logs are not stored, you cannot view them after the build completes. The fix is to store logs in a persistent store (e.g., a database or object storage). The fifth pitfall is search. Finding a specific log line in thousands of lines is hard without search. The fix is to implement search (e.g., client-side search for recent logs, server-side search for historical logs).
Advanced Build Log Streaming Patterns
Beyond the basics, the build log streaming system supports several advanced patterns. The first is log search. For builds that generate thousands of log lines, finding a specific line (e.g., the error that caused the build to fail) is hard without search. The system supports client-side search (for recent logs in the dashboard) and server-side search (for historical logs stored in the persistent store). The search supports regular expressions, which means you can search for patterns (e.g., Error:.*module to find all module errors).
The second pattern is log filtering. The system supports log filtering by level (e.g., show only ERROR and WARN lines) and by source (e.g., show only lines from the TypeScript compiler, not from the bundler). This is useful for builds with verbose output, where the important lines are buried in noise.
The third pattern is log export. The system supports log export (e.g., download the full log as a text file), which is useful for sharing with team members or for archiving. The exported log includes all lines (not just the ones visible in the dashboard), with timestamps and log levels.
The fourth pattern is log comparison. For debugging build failures, it is useful to compare the log from a failed build with the log from a successful build. The system supports log comparison (diff), which highlights the differences between two builds. This makes it easy to identify what changed and caused the failure.
The fifth pattern is log sharing. The system supports log sharing (e.g., a shareable URL that shows the build log), which is useful for collaborating with team members or for reporting bugs. The shared URL is read-only and expires after a configurable period (e.g., 7 days).
How Build Log Streaming Integrates with the MCP Server
The build log streaming is exposed via the Deployxa MCP server as the deployxa_get_build_log tool. This means your AI assistant (in Cursor or Claude Desktop) can fetch build logs directly and reason about them. For example, if a deployment fails, you can say "show me the build log and tell me what went wrong," and your AI assistant calls deployxa_get_build_log, reads the log, identifies the error, and proposes a fix. This is the agentic debugging workflow, where the AI assistant diagnoses build failures without you leaving your editor. For more on the MCP server, see our article on giving Cursor cloud superpowers. For more on debugging, see our article on debugging from your IDE with deployxa doctor.
Lessons Learned
Building the build log streaming system taught us several lessons. First, real-time is essential for developer experience. Seeing build logs stream in real time gives developers immediate feedback, which means they can catch and fix issues quickly. Without real-time streaming, developers would have to wait for the build to complete and then read the full log, which is slower and less productive. Second, batching is necessary for scalability. Sending each log line individually (via WebSocket) is inefficient for builds that generate thousands of lines. Batching log lines (sending them in groups) reduces the overhead per line and makes the system scalable. Third, the persistent store is essential for debugging. Without a persistent store, logs are lost when the build completes, which means you cannot review them later. With a persistent store, logs are available for historical queries, which is essential for debugging intermittent issues. Fourth, log search is essential for usability. Builds can generate thousands of lines, and finding the important line (e.g., the error) without search is like finding a needle in a haystack. With search, finding the important line is fast. Fifth, the WebSocket reconnection is critical for reliability. WebSocket connections can drop (due to network issues, server restarts, etc.), and without reconnection, the client misses log lines. With reconnection (and sending missed lines from the persistent store), the client always has the complete log. For more on Deployxa's engineering, see our articles on how we built the logging pipeline and how we built the CI/CD pipeline.
Conclusion: Real-Time Build Logs for Immediate Feedback
Deployxa's build log streaming system delivers build logs to your browser in real time, with sub-second latency. The four components (build process, log collector, WebSocket server, dashboard) work together to give you immediate feedback on your build's progress. For more on Deployxa's engineering, see our articles on the Git integration system and how we handle DDoS protection. Learn about the container image registry and how we built the CI/CD pipeline in our companion articles. Explore our free developer tools. Try Deployxa Drop for an instant live preview.