Google+

Shineyrock

web design & consultancy

Shineyrock

  1. web design and consultancy

blog

  1. Home
  2. blog
  3. read
  4. Deferring Tasks in Laravel Using Queues
  • dislike -3 13

    Deferring Tasks in Laravel Using Queues

    1. November 2017
    2. Daily
    3. read 560

    In this article, we're going to explore the Queue API in the Laravel web framework. It allows you to defer resource-intensive tasks during script execution to enhance the overall end user experience. After introducing the basic terminology, I'll demonstrate it by implementing a real-world example.

    Page load time is an important aspect of any successful website as it affects both the SEO of the site and the overall user experience. More often than not, you end up looking to debug web pages with long page load times. Of course, there are different approaches you could use to rectify this issue.

    Upon investigation, you often realize that certain code blocks are causing a delay in the page execution. The next thing you could try is identifying blocks that can be deferred for processing and that have no real impact on the end result of the current page. That should improve the overall web page speed as we've eliminated code blocks that were causing a delay.

    Today, we're going to explore a similar concept in the context of the Laravel web framework. In fact, Laravel already provides a useful built-in API that allows us to defer the processing of tasks—the Queue API. Without wasting much of your time, I'll go ahead and discuss the basic elements of the Queue API.

    Drivers, Connections, Queues, and Jobs

    The basic purpose of the Queue API is to run jobs that are added in a queue. Next, the queue could belong to a specific connection, and that connection may belong to a specific queue driver configured with that connection itself. Let's briefly try to understand what I've just said.

    Queue Drivers

    In the same way you would have used a different driver for your database connection, you could also choose from a variety of different queue drivers. The Queue API supports different adapters like database, beanstalkd, sqs, and redis.

    The queue driver is just a place that is used to store queue-related information. So if you're using a database queue driver, for example, the new job will be added in the jobs table in the database. On the other hand, if you've configured redis as the default queue driver, the job will be added to the redis server.

    The Queue API also provides two special queue drivers for testing purposes—sync and null. The sync queue driver is used to execute a queue job immediately, while the null queue driver is used to skip a job so that it won't be executed at all.

    Connections

    When you configure the Queue API for the first time, you need to specify a default connection that should be used for default queue processing. At the very least, the connection is expected to provide the following information:

    • the queue driver that will be used
    • the queue driver's specific configuration values
    • the default queue name in which the job will be added

    Queues

    When you add any job into a queue, it'll be added into the default queue. That should be fine in most cases, unless you have jobs that need to be given higher priority over other jobs. In that case, you could create a queue named high and place the higher priority jobs in that particular queue.

    When you run a queue worker that processes queued jobs, you could optionally pass the --queue parameter, which allows you to list queue names in the order in which they need to be processed. For example, if you specify --queue=high,default, it will first process jobs in the high queue, and once it's completed it fetches jobs in the default queue.

    Jobs

    A job in the Queue API is a task that's deferred from the main execution flow. For example, if you want to create a thumbnail when the user uploads an image from the front-end, you could create a new job that handles the thumbnail processing. In this way, you could defer the task of thumbnail processing from the main execution flow.

    That was a basic introduction to the Queue API terminology. From the next section onwards, we'll explore how to create a custom queue job and run it by using a Laravel queue worker.

    A Quick Look at the Queue Configuration File

    In the previous sections, we've gone through the terminology of queues in Laravel. Let's have a quick look at the config/queue.php configuration file so that you can relate it to the concepts that we've discussed so far.

    <?php

return [

    /*
    |--------------------------------------------------------------------------
    | Default Queue Connection Name
    |--------------------------------------------------------------------------
    |
    | Laravel's queue API supports an assortment of back-ends via a single
    | API, giving you convenient access to each back-end using the same
    | syntax for every one. Here you may define a default connection.
    |
    */

    'default' => env('QUEUE_CONNECTION', 'sync'),

    /*
    |--------------------------------------------------------------------------
    | Queue Connections
    |--------------------------------------------------------------------------
    |
    | Here you may configure the connection information for each server that
    | is used by your application. A default configuration has been added
    | for each back-end shipped with Laravel. You are free to add more.
    |
    | Drivers: "sync", "database", "beanstalkd", "sqs", "redis", "null"
    |
    */

    'connections' => [

        'sync' => [
            'driver' => 'sync',
        ],

        'database' => [
            'driver' => 'database',
            'table' => 'jobs',
            'queue' => 'default',
            'retry_after' => 90,
        ],

        'beanstalkd' => [
            'driver' => 'beanstalkd',
            'host' => 'localhost',
            'queue' => 'default',
            'retry_after' => 90,
            'block_for' => 0,
        ],

        'sqs' => [
            'driver' => 'sqs',
            'key' => env('AWS_ACCESS_KEY_ID'),
            'secret' => env('AWS_SECRET_ACCESS_KEY'),
            'prefix' => env('SQS_PREFIX', 'https://sqs.us-east-1.amazonaws.com/your-account-id'),
            'queue' => env('SQS_QUEUE', 'your-queue-name'),
            'suffix' => env('SQS_SUFFIX'),
            'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
        ],

        'redis' => [
            'driver' => 'redis',
            'connection' => 'default',
            'queue' => env('REDIS_QUEUE', 'default'),
            'retry_after' => 90,
            'block_for' => null,
        ],

    ],

    /*
    |--------------------------------------------------------------------------
    | Failed Queue Jobs
    |--------------------------------------------------------------------------
    |
    | These options configure the behavior of failed queue job logging so you
    | can control which database and table are used to store the jobs that
    | have failed. You may change them to any database / table you wish.
    |
    */

    'failed' => [
        'driver' => env('QUEUE_FAILED_DRIVER', 'database'),
        'database' => env('DB_CONNECTION', 'mysql'),
        'table' => 'failed_jobs',
    ],

];

    As you can see, you can choose from several different sets of connections. For each connection, you can configure the connection-specific information, queue driver, and default queue name. More importantly, you can also set the default queue connection which you want to use under the default key.

    So that's the basics of the queue configuration file.

    Create Your First Queue Job

    By now, you should feel confident about queue jobs. From this section onwards, we're going to implement a real-world example that demonstrates the concept of queue jobs in Laravel.

    You'll often need to create different thumbnail versions of an image uploaded by a user. In most cases, the developer tries to process it in real time so that different versions of images are created right away when the user uploads an image.

    It seems to be a reasonable approach if you're going to create a couple of versions and it doesn't take too much time in the first place. On the other hand, if you're dealing with an application that requires heavy processing and thus eats up more resources, real-time processing could end up in a bad user experience.

    The obvious option that first pops up in your mind is to defer processing of the thumbnail generation as late as possible. The simplest approach you could implement in this specific scenario is to set a cron job that triggers processing at regular intervals, and you should be fine.

    A much better approach, on the other hand, is to defer and push the task into a queue, and let the queue worker process it when it gets a chance to do so. In a production environment, the queue worker is a daemon script that's always running and processing tasks in a queue. The obvious benefit of this approach is a much better end user experience, and you don't have to wait for the cron run as the job will be processed as soon as possible.

    I guess that's enough theory to get started with an actual implementation.

    Create the jobs Table

    In our case, we're going to use the database queue driver, and it requires us to create the jobs table in the database. The jobs table holds all the jobs that need to be processed in the next queue worker run.

    Before we go ahead and create the jobs table, let's change the default queue configuration from sync to database in the .env file.

    ...
...
QUEUE_CONNECTION=database
...
...

    In fact, Laravel already provides an artisan command that helps us to create the jobs table. Run the following command in the root of your Laravel application, and it should create the necessary database migration that creates the jobs table.

    $php artisan queue:table

    The migration file which is generated at database/migrations/YYYY_MM_DD_HHMMSS_create_jobs_table.php should look like this:

    <?php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

class CreateJobsTable extends Migration
{
    /**
     * Run the migrations.
     *
     * @return void
     */
    public function up()
    {
        Schema::create('jobs', function (Blueprint $table) {
            $table->bigIncrements('id');
            $table->string('queue')->index();
            $table->longText('payload');
            $table->unsignedTinyInteger('attempts');
            $table->unsignedInteger('reserved_at')->nullable();
            $table->unsignedInteger('available_at');
            $table->unsignedInteger('created_at');
        });
    }

    /**
     * Reverse the migrations.
     *
     * @return void
     */
    public function down()
    {
        Schema::dropIfExists('jobs');
    }
}

    Next, let's run the migrate command so that it actually creates the jobs table in a database.

    $php artisan migrate

    That's it as far as the jobs migration is concerned.

    Create the Image Model

    Next, let's create the Image model that will be used to manage images uploaded by the end user. The image model also requires an associated database table, so we'll use the --migrate option while creating the Image model.

    $php artisan make:model Image --migration

    The above command should create the Image model class and an associated database migration as well.

    The Image model class should look like this:

    <?php

namespace App;

use Illuminate\Database\Eloquent\Model;

class Image extends Model
{
    //
}

    And the database migration file should be created at database/migrations/YYYY_MM_DD_HHMMSS_create_images_table.php. We also want to store the original path of the image uploaded by the end user. Let's revise the code of the Image database migration file to look like the following.

    <?php
// database/migrations/YYYY_MM_DD_HHMMSS_create_images_table.php
use Illuminate\Support\Facades\Schema;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Database\Migrations\Migration;

class CreateImagesTable extends Migration
{
    /**
     * Run the migrations.
     *
     * @return void
     */
    public function up()
    {
        Schema::create('images', function (Blueprint $table) {
            $table->increments('id');
            $table->timestamps();
            $table->string('org_path');
        });
    }

    /**
     * Reverse the migrations.
     *
     * @return void
     */
    public function down()
    {
        Schema::dropIfExists('images');
    }
}

    As you can see, we've added the $table->string('org_path') column to store the path of the original image. Next, you just need to run the migrate command to actually create that table in the database.

    php artisan migrate

    And that's it as far as the Image model is concerned.

    Create a Laravel Job

    Next, let's create an actual queue job that's responsible for processing image thumbnails. For the thumbnail processing, we're going to use a very popular image processing library—Intervention Image.

    To install the Intervention Image library, go ahead and run the following command at the root of your application.

    php composer.phar require intervention/image

    Now, it's time to create the Job class, and we'll use an artisan command to do that.

    php artisan make:job ProcessImageThumbnails

    That should create the Job class template at app/Jobs/ProcessImageThumbnails.php. Let's replace the contents of that file with the following.

    <?php
// app/Jobs/ProcessImageThumbnails.php
namespace App\Jobs;

use App\Image as ImageModel;
use Illuminate\Bus\Queueable;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Support\Facades\DB;

class ProcessImageThumbnails implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    protected $image;

    /**
     * Create a new job instance.
     *
     * @return void
     */
    public function __construct(ImageModel $image)
    {
        $this->image = $image;
    }

    /**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {
        // access the model in the queue for processing
        $image = $this->image;
        $full_image_path = public_path($image->org_path);
        $resized_image_path = public_path('thumbs' . DIRECTORY_SEPARATOR .  $image->org_path);

        // create image thumbs from the original image
        $img = \Image::make($full_image_path)->resize(300, 200);
        $img->save($resized_image_path);
    }
}

    When the queue worker starts processing any job, it looks for the handle method. So it's the handle method that holds the main logic of your job.

    In our case, we need to create a thumbnail of an image uploaded by the user. The code of the handle method is pretty straightforward—we retrieve an image from the ImageModel model and create a thumbnail using the Intervention Image library. Of course, we need to pass the corresponding Image model when we dispatch our job, and we'll see it in a moment.

    How to Test a Job

    To test our newly created job, we'll create a simple upload form that allows the user to upload an image. Of course, we won't create image thumbnails right away; we'll defer that task so that it can be processed by the queue worker.

    Let's create a controller file at app/Http/Controllers/ImageController.php as shown below.

    <?php
namespace App\Http\Controllers;

use App\Image;
use App\Jobs\ProcessImageThumbnails;
use Illuminate\Http\Request;
use Illuminate\Support\Facades\Redirect;
use App\Http\Controllers\Controller;
use Validator;

class ImageController extends Controller
{
    /**
     * Show Upload Form
     *
     * @param  Request  $request
     * @return Response
     */
    public function index(Request $request)
    {
        return view('upload_form');
    }

    /**
     * Upload Image
     *
     * @param  Request  $request
     * @return Response
     */
    public function upload(Request $request)
    {
        // upload image
        $this->validate($request, [
          'demo_image' => 'required|image|mimes:jpeg,png,jpg,gif,svg|max:2048',
        ]);
        $image = $request->file('demo_image');
        $input['demo_image'] = time().'.'.$image->getClientOriginalExtension();
        $destinationPath = public_path('/images');
        $image->move($destinationPath, $input['demo_image']);

        // make db entry of that image
        $image = new Image;
        $image->org_path = 'images' . DIRECTORY_SEPARATOR . $input['demo_image'];
        $image->save();

        // defer the processing of the image thumbnails
        ProcessImageThumbnails::dispatch($image);

        return Redirect::to('image/index')->with('message', 'Image uploaded successfully!');
    }
}

    Let's create an associated view file at resources/views/upload_form.blade.php.

    <!DOCTYPE html>
<html lang="{{ config('app.locale') }}">
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <meta name="csrf-token" content="{{ csrf_token() }}" />
        <title>Laravel</title>

        <!-- Fonts -->
        <link href="https://fonts.googleapis.com/css?family=Raleway:100,600" rel="stylesheet" type="text/css">

        <!-- Styles -->
        <style>
            html, body {
                background-color: #fff;
                color: #636b6f;
                font-family: 'Raleway', sans-serif;
                font-weight: 100;
                height: 100vh;
                margin: 0;
            }

            .full-height {
                height: 100vh;
            }

            .flex-center {
                align-items: center;
                display: flex;
                justify-content: center;
            }

            .position-ref {
                position: relative;
            }

            .top-right {
                position: absolute;
                right: 10px;
                top: 18px;
            }

            .content {
                text-align: center;
            }

            .title {
                font-size: 84px;
            }

            .links > a {
                color: #636b6f;
                padding: 0 25px;
                font-size: 12px;
                font-weight: 600;
                letter-spacing: .1rem;
                text-decoration: none;
                text-transform: uppercase;
            }

            .m-b-md {
                margin-bottom: 30px;
            }

            .alert {
                color: red;
                font-weight: bold;
                margin: 10px;
            }
            .success {
                color: blue;
                font-weight: bold;
                margin: 10px;
            }
        </style>
    </head>
    <body>
        <div class="flex-center position-ref full-height">
            @if (Route::has('login'))
                <div class="top-right links">
                    @if (Auth::check())
                        <a href="{{ url('/home') }}">Home</a>
                    @else
                        <a href="{{ url('/login') }}">Login</a>
                        <a href="{{ url('/register') }}">Register</a>
                    @endif
                </div>
            @endif

            <div class="content">
                <div class="m-b-md">
                    <h1 class="title">Demo Upload Form</h1>

                    @if ($errors->any())
                        <div class="alert alert-danger">
                            <ul>
                                @foreach ($errors->all() as $error)
                                    <li>{{ $error }}</li>
                                @endforeach
                            </ul>
                        </div>
                    @endif

                    @if (session('message'))
                        <div class="success">
                            {{ session('message') }}
                        </div>
                    @endif

                    <form method="post" action="{{ url('/image/upload') }}" enctype="multipart/form-data">
                      <div>
                        <input type="file" name="demo_image" />
                      </div>
                      <br/>
                      <div>
                        <input type="hidden" name="_token" value="{{ csrf_token() }}">
                        <input type="submit" value="Upload Image"/>
                      </div>
                    </form>
                </div>
            </div>
        </div>
    </body>
</html>

    Finally, let's add routes for the index and upload actions in the routes/web.php file.

    Route::get('image/index', 'ImageController@index');
Route::post('image/upload', 'ImageController@upload');

    In the ImageController controller, the index method is used to render an upload form.

    public function index(Request $request)
{
    return view('upload_form');
}

    When the user submits a form, the upload method is invoked.

    public function upload(Request $request)
{
    // upload image
    $this->validate($request, [
        'demo_image' => 'required|image|mimes:jpeg,png,jpg,gif,svg|max:2048',
    ]);
    $image = $request->file('demo_image');
    $input['demo_image'] = time().'.'.$image->getClientOriginalExtension();
    $destinationPath = public_path('/images');
    $image->move($destinationPath, $input['demo_image']);

    // make db entry of that image
    $image = new Image;
    $image->org_path = 'images' . DIRECTORY_SEPARATOR . $input['demo_image'];
    $image->save();

    // defer the processing of the image thumbnails
    ProcessImageThumbnails::dispatch($image);

    return Redirect::to('image/index')->with('message', 'Image uploaded successfully!');
}

    At the beginning of the upload method, you'll notice the usual file upload code that moves the uploaded file to the public/images directory. Next, we insert a database record using the App/Image model.

    Finally, we use the ProcessImageThumbnails job to defer the thumbnail processing task. It's important to note that it's the dispatch method that's used to defer a task. At the end, the user is redirected to the upload page with a success message.

    At this point in time, the job is added to the jobs table for processing. Let's confirm it by issuing the following query.

    mysql> select * FROM jobs;
|  1 | default | {"displayName":"App\\Jobs\\ProcessImageThumbnails","job":"Illuminate\\Queue\\CallQueuedHandler@call","maxTries":null,"timeout":null,"data":{"commandName":"App\\Jobs\\ProcessImageThumbnails","command":"O:31:\"App\\Jobs\\ProcessImageThumbnails\":5:{s:8:\"\u0000*\u0000image\";O:45:\"Illuminate\\Contracts\\Database\\ModelIdentifier\":2:{s:5:\"class\";s:9:\"App\\Image\";s:2:\"id\";i:2;}s:6:\"\u0000*\u0000job\";N;s:10:\"connection\";N;s:5:\"queue\";N;s:5:\"delay\";N;}"}} |        0 |        NULL |   1510219099 | 1510219099 |

    You must be wondering, what does it take to process a job then? Don't worry—that's what we're going to discuss in the very next section.

    Queue Worker

    The job of the Laravel queue worker is to process jobs that are queued up for processing. In fact, there's an artisan command that helps us to start the queue worker process.

    $php artisan queue:work

    As soon as you run that command, it processes pending jobs. In our case, it should process the ProcessImageThumbnails job that was queued up when the user uploaded an image earlier.

    $php artisan queue:work
[YYYY-MM-DD HHMMSS] Processing: App\Jobs\ProcessImageThumbnails
[YYYY-MM-DD HHMMSS] Processed:  App\Jobs\ProcessImageThumbnails

    You would have noticed that when you start a queue worker, it keeps running until you kill it manually or close the terminal. In fact, it's waiting for the next job to be processed. As soon as there's a new job in the queue, it'll be processed right away if the queue worker is running.

    Of course, we can't keep it running that way, so we need to find a way for the queue worker to run permanently in the background.

    To our rescue, there are several process management tools out there you could choose from. To name a few, here's a list:

    • Circus
    • daemontools
    • Monit
    • Supervisor
    • Upstart

    You should choose a tool that you're comfortable with to manage the Laravel queue worker. Basically, we want to make sure that the queue worker should run indefinitely so that it processes queued jobs right away.

    So that's the Queue API at your disposal. You can use it in your day-to-day development to defer time-consuming tasks for the betterment of the end user experience.

    Conclusion

    In this article, we discussed the Queue API in Laravel, which is really helpful should you wish to defer processing of resource-consuming tasks.

    We started with a basic introduction to the Queue API, which involved a discussion of connections, queues, and jobs. In the second half of the article, we created a custom queue job that demonstrated how you could use the Queue API in the real world.

    For those of you who are either just getting started with Laravel or looking to expand your knowledge, site, or application with extensions, we have a variety of things you can study at Envato Market.

    Photo

    martijn broeders

    founder/ strategic creative at shineyrock web design & consultancy
    e-mail: .(JavaScript must be enabled to view this email address)
    phone: 434 210 0245
    Add a comment...
    Comments

By - category

By - date

Copyright © 1999 - 2026 Shineyrock web design and consultancy, All Rights Reserved / Terms of Conditions / Sitemap: XML, HTML, Images, Mobile, News