Debugging a Dockerized Rails ERP System: A Personal Journey Through Configuration Layers

The Initial Discovery

My Friday afternoon took an unexpected turn when I attempted to log into my company's dockerized ERP system. Instead of the familiar dashboard, I was greeted with a stark "Network Error" message. What followed was a debugging journey that took me through every layer of modern web application architecture.

The browser console revealed the first clue:

Failed to load resource: the server responded with a status of 500 (Internal Server Error)
Refused to connect to 'http://localhost:3000/api/v1/auth/login' because it violates the Content Security Policy

Layer 1: Tracing the Network Configuration

My first instinct was to verify the Docker deployment. I needed to understand what was actually running:

docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Ports}}\t{{.Status}}"

The output revealed my architecture:

pcvn-erp-frontend-prod   pcvn-fullstack_frontend   0.0.0.0:3003->80/tcp    Up 10 minutes (healthy)
pcvn-erp-backend-prod    pcvn-fullstack_backend    0.0.0.0:3002->3002/tcp  Up 10 minutes (healthy)
pcvn-erp-db-prod         postgres:15-alpine        0.0.0.0:5432->5432/tcp  Up 10 minutes (healthy)

I immediately spotted the discrepancy - my frontend was trying to reach port 3000, but the backend was running on port 3002. The Content Security Policy was correctly blocking the non-existent port 3000, but even if it hadn't, there was nothing listening there.

Investigating the Frontend Configuration

I needed to understand where this port 3000 was coming from. I searched the compiled JavaScript in the frontend container:

docker exec pcvn-erp-frontend-prod grep -r "localhost:3000" /usr/share/nginx/html/assets/

The search confirmed my suspicion - the API URL was hardcoded in the JavaScript bundles as VITE_API_URL:"http://localhost:3000/api/v1". This meant the misconfiguration was baked into the build, not a runtime setting I could easily change.

Layer 2: Rebuilding with Correct Configuration

I examined my docker-compose.prod.yml and found it had the correct default:

VITE_API_URL: ${VITE_API_URL:-http://localhost:3002/api/v1}

The frontend image must have been built with an older configuration. I needed a complete rebuild:

docker-compose -f docker-compose.prod.yml build --no-cache frontend

After rebuilding, I recreated the entire stack to resolve a network configuration change:

docker-compose -f docker-compose.prod.yml down
docker-compose -f docker-compose.prod.yml up -d

I verified the fix by checking the new JavaScript bundles:

docker exec pcvn-erp-frontend-prod grep -o "localhost:30[0-9][0-9]" /usr/share/nginx/html/assets/index-*.js | sort | uniq -c

Result:

6 localhost:3002
2 localhost:3003

Perfect - no more references to port 3000.

Layer 3: Database Connection Mysteries

With the network issue resolved, I encountered a new error: "Request failed with status code 500". The backend logs revealed:

docker logs pcvn-erp-backend-prod --tail 50

ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "users" does not exist

My database tables hadn't been created. I attempted to run migrations:

docker exec pcvn-erp-backend-prod bundle exec rails db:migrate

This failed with a connection error - Rails was trying to connect to localhost instead of the database container. I discovered Rails was looking for DATABASE_HOST environment variable, which wasn't set:

docker exec pcvn-erp-backend-prod printenv DATABASE_HOST
# (empty response)

Understanding the Multi-Database Configuration

Examining the database.yml revealed a sophisticated multi-database setup:

docker exec pcvn-erp-backend-prod grep -A 20 "^production:" config/database.yml

Rails expected four databases: primary, cache, queue, and cable. The configuration used individual environment variables rather than parsing the DATABASE_URL I had set.

Layer 4: PostgreSQL User Permissions

I attempted migrations with the correct environment variables:

docker exec -e DATABASE_HOST=db -e BACKEND_DATABASE_PASSWORD=pcvn_prod_db_2025_secure_password_8f3k9m2p pcvn-erp-backend-prod bundle exec rails db:migrate

This revealed another layer - Rails expected a 'backend' user, but my PostgreSQL only had 'pcvn'. I created the missing user:

docker exec -it pcvn-erp-db-prod psql -U pcvn -d pcvn_erp

CREATE USER backend WITH PASSWORD 'pcvn_prod_db_2025_secure_password_8f3k9m2p';
ALTER USER backend CREATEDB;
GRANT ALL PRIVILEGES ON DATABASE pcvn_erp TO backend;
GRANT ALL ON SCHEMA public TO backend;

Layer 5: The Permission Paradox

Even after creating the user, I still got "relation does not exist" errors. Investigating further:

docker exec pcvn-erp-db-prod psql -U backend -d pcvn_erp -c "\dt"

The tables existed! But they were owned by 'pcvn', not 'backend'. I needed to grant permissions on existing tables:

GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO backend;
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO backend;

Layer 6: Creating the First User

With all infrastructure issues resolved, I still couldn't log in - the database had no user records. I created an admin account through Rails console:

docker exec -it -e DATABASE_HOST=db -e BACKEND_DATABASE_PASSWORD=pcvn_prod_db_2025_secure_password_8f3k9m2p pcvn-erp-backend-prod bundle exec rails console

# First, create the admin role
admin_role = Role.create!(name: 'admin', description: 'System Administrator')

# Then create the user
User.create!(
  email: 'admin@pcvn.com', 
  password: 'SecurePassword123!', 
  password_confirmation: 'SecurePassword123!', 
  role: admin_role
)

Reflection on the Journey

This debugging session reinforced several lessons I've learned about containerized applications:

1. Configuration happens at multiple layers - Build-time variables get baked into images, while runtime variables can be changed. Understanding which is which is crucial.

2. Error messages don't always point to root causes - "Network Error" was actually a port misconfiguration. "Table doesn't exist" was actually a permissions issue.

3. Docker networking adds complexity - Each container has its own network namespace. 'localhost' means different things in different contexts.

4. Database permissions are nuanced - Having database access doesn't mean having table access. User creation doesn't automatically grant permissions on existing objects.

5. Systematic investigation beats random fixes - By methodically checking each layer, I avoided making unnecessary changes and understood the actual problems.

The entire issue chain was:

• Frontend → Wrong port in compiled JavaScript
• Backend → Missing database user
• Database → Missing permissions on tables
• Application → Missing user records

Each fix revealed the next issue, like archaeology through the layers of a modern web application. What started as a simple login error became a masterclass in debugging distributed systems. The experience reminded me that patience and systematic investigation are the most valuable debugging tools - more important than any specific command or technique.

Technical Commands Reference

For future reference, these commands proved invaluable:

# Docker investigation
docker ps --format "table {{.Names}}\t{{.Ports}}\t{{.Status}}"
docker logs [container-name] --tail 50
docker exec [container] printenv | grep [PATTERN]

# File investigation in containers
docker exec [container] ls -la [path]
docker exec [container] grep -r [pattern] [path]

# Rails debugging
docker exec [container] bundle exec rails db:migrate
docker exec [container] bundle exec rails db:migrate:status
docker exec [container] bundle exec rails console

# PostgreSQL management
docker exec -it [db-container] psql -U [user] -d [database]
\dt  # List tables
\q   # Exit

The hours invested in understanding these issues thoroughly means I now have a robust, properly configured system and deep knowledge of its architecture. Sometimes the longest debugging sessions provide the most valuable learning experiences.

If you enjoyed this article, you can also find it published on LinkedIn and Medium.