Fixing Slow CSV Imports with Sidekiq Background Jobs

Fixing Slow CSV Imports with Sidekiq Background Jobs

Hey everyone! 👋

So I ran into this annoying problem recently that I bet many of you have faced. I was building a tournament management app, and needed to let organizers bulk import player lists from CSV files. Sounds simple, right?

Wrong. SO wrong.

The Problem: Blocking the Entire Request

Here’s what was happening: An organizer would upload a CSV with 500+ players, hit submit, and then… wait. And wait. And wait some more.

The page would just hang there loading. Meanwhile, the entire Rails process was stuck processing the CSV file, inserting records one by one. If they tried to refresh or navigate away? Everything broke. Lost progress. Angry users. Not good.

# app/controllers/players_controller.rb
class PlayersController < ApplicationController
  def import
    csv_file = params[:file]

    # This blocks the entire request thread
    CSV.foreach(csv_file.path, headers: true) do |row|
      Player.create!(
        tournament_id: params[:tournament_id],
        name: row['name'],
        email: row['email'],
        deck_name: row['deck']
      )
    end

    redirect_to tournament_players_path, notice: "Players imported successfully!"
  rescue StandardError => e
    redirect_to tournament_players_path, alert: "Import failed: #{e.message}"
  end
end

The worst part? While this import was running, the user couldn’t even navigate to other pages because the browser was waiting for this request to complete. Total UX disaster.

Enter Sidekiq: Let the Background Do the Heavy Lifting

The solution was actually pretty simple once I wrapped my head around it: move the work to a background job. That way:

1. User uploads CSV
2. We immediately respond with a success message
3. The actual import happens in the background
4. User can continue using the app while it processes

Sidekiq is perfect for this because it’s fast, reliable, and dead simple to set up.

Setting Up Sidekiq

First, let’s add Sidekiq to our Gemfile:

# Gemfile
gem 'sidekiq', '~> 7.1'
gem 'redis', '~> 5.0'

bundle install

Then configure Sidekiq in your Rails app:

# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
  config.redis = { url: ENV['REDIS_URL'] || 'redis://localhost:6379/0' }
end

Sidekiq.configure_client do |config|
  config.redis = { url: ENV['REDIS_URL'] || 'redis://localhost:6379/0' }
end

Creating the Background Job

Now let’s create a Sidekiq worker to handle the CSV processing:

# app/workers/player_import_worker.rb
class PlayerImportWorker
  include Sidekiq::Worker

  sidekiq_options retry: 3, queue: :default

  def perform(tournament_id, file_path)
    tournament = Tournament.find(tournament_id)
    success_count = 0
    error_count = 0
    players_batch = []

    CSV.foreach(file_path, headers: true).each_slice(100) do |rows|
      rows.each do |row|
        players_batch << {
          tournament_id: tournament_id,
          name: row['name'],
          email: row['email'],
          deck_name: row['deck'],
          created_at: Time.current,
          updated_at: Time.current
        }
      end

      begin
        # Bulk insert the batch
        Player.insert_all(players_batch)
        success_count += players_batch.size
      rescue StandardError => e
        Rails.logger.error "Failed to import batch: #{e.message}"
        error_count += players_batch.size
      end

      players_batch.clear
    end

    # Clean up the temp file
    File.delete(file_path) if File.exist?(file_path)

    Rails.logger.info "Import complete: #{success_count} successful, #{error_count} failed"
  rescue StandardError => e
    Rails.logger.error "Import job failed: #{e.message}"
    raise
  end
end

Updating the Controller

Now we update the controller to just queue the job instead of doing the work:

# app/controllers/players_controller.rb
class PlayersController < ApplicationController
  def import
    csv_file = params[:file]
    tournament_id = params[:tournament_id]

    # Save the file to a temporary location
    temp_path = Rails.root.join('tmp', 'imports', "#{SecureRandom.uuid}.csv")
    FileUtils.mkdir_p(File.dirname(temp_path))
    FileUtils.cp(csv_file.path, temp_path)

    # Queue the background job
    PlayerImportWorker.perform_async(tournament_id, temp_path.to_s)

    # Immediately respond to the user
    redirect_to tournament_players_path,
                notice: "Import started! Players will appear shortly."
  rescue StandardError => e
    redirect_to tournament_players_path,
                alert: "Failed to start import: #{e.message}"
  end
end

Running Sidekiq

To actually process the jobs, you need to run the Sidekiq process:

bundle exec sidekiq

In production, you’d want to use a process manager like systemd or run it as a separate container if you’re using Docker.

Monitoring with the Sidekiq Web UI

One of the coolest things about Sidekiq is the built-in web UI. You can see all your jobs, retry failed ones, and monitor performance.

Add this to your routes:

# config/routes.rb
require 'sidekiq/web'

Rails.application.routes.draw do
  mount Sidekiq::Web => '/sidekiq'

  # ... rest of your routes
end

Now visit http://localhost:3000/sidekiq and you’ll see a beautiful dashboard:

The dashboard shows:

• How many jobs are queued, processing, or failed
• Real-time stats on job processing
• The ability to retry failed jobs manually
• Memory usage and performance metrics

Pro tip: In production, make sure to protect this route with authentication! You don’t want anyone accessing your job dashboard.

The Result: Night and Day Difference

After implementing Sidekiq, the user experience was SO much better:

Before:

• Upload CSV → Page hangs for 2+ minutes → Hope nothing times out
• Can’t navigate away or do anything else
• Any error loses all progress

After:

• Upload CSV → Instant redirect with confirmation
• Can immediately browse other pages
• Job retries automatically if something fails
• Players appear in the list as they’re imported

What I Learned

1. Don’t block the request thread - If something takes more than a second or two, it should probably be a background job

2. Redis is your friend - Sidekiq uses Redis as a queue, and it’s crazy fast. Installation is simple too

3. Always clean up temp files - Make sure your workers delete temporary files when done

4. Error handling matters - Log errors properly so you can debug issues later

5. User feedback is key - Even though processing happens in the background, users still need to know what’s happening

Optional: Adding Progress Updates

Want to get fancy? You can even add progress updates using Action Cable or polling. But honestly, for most cases, just showing “Import started!” is good enough.

Resources

• Sidekiq Documentation
• Redis Quick Start
• Rails Background Jobs Guide

Pro tip: If you’re not ready to add Redis to your stack, check out ActiveJob with the built-in Async adapter for simpler cases. But honestly, Sidekiq is worth it!