Debugging a Rails 8 Sidekiq Container: A Production Troubleshooting Journey

The Initial Problem

What started as a seemingly simple Docker container restart issue turned into an extensive debugging journey through a Rails 8 application's architecture. My Sidekiq container was stuck in an endless restart loop, repeatedly failing with exit code 127. The Docker Desktop logs showed a cascade of errors, all stemming from what appeared to be a straightforward problem: bundler: command not found: sidekiq.

The Systematic Debugging Approach

1. Missing Sidekiq Gem

Problem: The container couldn't find the sidekiq executable.

Discovery Command:

docker run --rm pcvn-fullstack-sidekiq:latest bundle list | grep sidekiq

Root Cause: The Sidekiq gem was completely absent from the Gemfile.

Solution: Added gem "sidekiq", "~> 7.3" to the Gemfile and ran bundle install.

2. Missing BaseConnector Class

Problem: After adding Sidekiq, the application failed with uninitialized constant Connectors::BaseConnector.

Discovery Command:

ls -la backend/app/connectors/

Root Cause: The SapConnector class was trying to inherit from a non-existent BaseConnector class.

Solution: Created the missing base_connector.rb file with appropriate base functionality for all connector classes.

3. Zeitwerk Autoloading Convention Issues

Problem: Rails expected BaseConnector at the root level, but I had initially wrapped it in a module Connectors.

Discovery Commands:

docker logs pcvn-erp-sidekiq-prod 2>&1 | head -100
grep -A 2 "class BaseConnector" /rails/app/connectors/base_connector.rb

Root Cause: Zeitwerk's autoloading expects files in app/connectors/ to define classes at the root level, not inside modules.

Solution: Removed the module wrapper, defining the class directly as class BaseConnector.

4. SapConnector Naming Mismatch

Problem: The file defined class SAPConnector but Zeitwerk expected class SapConnector.

Discovery Command:

head -10 /rails/app/connectors/sap_connector.rb

Root Cause: Rails' naming conventions require "Sap" not "SAP" for the class name when the file is named sap_connector.rb.

Solution: Renamed the class from SAPConnector to SapConnector and removed its module wrapper.

5. Ruby Syntax Errors in Hash Literals

Problem: Multiple files had syntax errors with rescue modifiers inside hash literals.

Discovery Pattern:

# This caused syntax errors:
performance_score: kpis[:summary][:performance_score] rescue nil

# Fixed with parentheses:
performance_score: (kpis[:summary][:performance_score] rescue nil)

Affected Files:

• analytics_controller.rb (line 31)
• ab_testing_service.rb (line 406)
• innovation_metrics_service.rb (line 231)

Discovery Command:

docker logs pcvn-erp-sidekiq-prod 2>&1 | grep -B5 "syntax error"

Solution: Wrapped all rescue expressions in parentheses to create grouped expressions.

6. Rails 8 Serialization API Changes

Problem: Multiple services used the old Rails serialization syntax with two arguments.

Discovery Error: wrong number of arguments (given 2, expected 1)

Affected Services:

• A/B Testing Service (4 serialize statements)
• Feature Flag Service (3 serialize statements)

Discovery Command:

grep -n "serialize :" backend/app/services/*.rb

Old Syntax:

serialize :metrics, JSON

New Rails 8 Syntax:

serialize :metrics, coder: JSON

Solution: Updated all serialize statements to use the coder: option required by Rails 8.

The Docker Build-Test Cycle

Each fix required rebuilding the Docker image and restarting the container:

# Rebuild with no cache to ensure all changes are included
docker build -t pcvn-fullstack-sidekiq:latest ./backend --no-cache

# Restart the container with the new image
docker-compose -f docker-compose.prod.yml down sidekiq
docker-compose -f docker-compose.prod.yml up -d sidekiq

# Check the logs for new errors or success
docker logs --tail 100 pcvn-erp-sidekiq-prod

Key Debugging Commands That Revealed Issues

# Check container status and restart count
docker ps -a --format "table {{.Names}}\t{{.Status}}\t{{.Image}}"

# Inspect what images containers are using
docker inspect pcvn-erp-sidekiq-prod --format='{{.Config.Image}}'

# Debug inside the container
docker exec -it pcvn-erp-sidekiq-prod /bin/bash

# Check Ruby syntax without running the file
ruby -c /rails/app/connectors/sap_connector.rb

# Search for patterns across files
grep -n "serialize :" backend/app/services/*.rb

Lessons Learned

Production Mode Reveals Hidden Issues

Rails' production mode eager loads all files at startup, revealing syntax errors and naming convention issues that might never surface in development mode's lazy loading. What initially felt frustrating became a comprehensive code audit that made my application more robust.

Cascading Errors Hide Deeper Problems

Each error I fixed revealed another issue that had been hidden behind it. The missing Sidekiq gem was just the first layer of a multi-layered problem that included missing classes, naming convention violations, syntax errors, and API compatibility issues.

Rails 8 API Evolution

The serialization API changes from Rails 7 to Rails 8 reflect a broader philosophy of making code more explicit and self-documenting. The new coder: syntax makes the developer's intent clearer, even if it requires updating existing code.

Docker Image Immutability

Understanding that Docker containers run from immutable images was crucial. Every code change required rebuilding the image - simply modifying files on my local system wouldn't affect the running container.

The Architecture Discovered

Through this debugging process, I discovered the sophisticated architecture of the ERP system I was working with:

• SAP Integration: Enterprise resource planning connectivity through OData protocols
• A/B Testing Service: Scientific experimentation platform for feature validation
• Feature Flag Service: Controlled rollout mechanism for safe deployments
• Innovation Metrics Service: Meta-analysis of experimentation effectiveness
• Background Job Processing: Sidekiq configuration with multiple priority queues and scheduled tasks

Final Success

After systematically addressing each issue, the Sidekiq container finally started successfully:

2025-08-16T19:41:45.410Z pid=1 tid=3i5 INFO: Booted Rails 8.0.2 application in production environment
2025-08-16T19:41:45.410Z pid=1 tid=3i5 INFO: Running in ruby 3.2.1
2025-08-16T19:41:45.411Z pid=1 tid=3i5 INFO: Sidekiq 7.3.9 connecting to Redis

This successful startup meant that Rails had completed its entire eager loading process, validating every file in the application. The background job processing infrastructure was now operational, ready to handle asynchronous tasks essential for the enterprise system's functionality.

Reflection

This debugging journey transformed what could have been a frustrating experience into a valuable learning experience in production system troubleshooting. I gained deep insights into Rails' autoloading system, Ruby's syntax requirements, Docker's architecture, and the importance of systematic debugging. Each error taught me something new about the application's structure and Rails' conventions.

Most importantly, I learned that production debugging isn't just about fixing immediate problems - it's about understanding the entire system's architecture and ensuring that every component follows best practices. The investment in properly fixing each issue rather than applying quick workarounds resulted in a more maintainable, robust application that's fully compatible with Rails 8's modern features.

The experience reinforced my appreciation for Rails' production mode eager loading. While it made the debugging process longer by revealing every issue upfront, it ultimately ensured that the application would run reliably in production without hidden surprises waiting to surface at the worst possible moment.

If you enjoyed this article, you can also find it published on LinkedIn and Medium.