Now that we have our batch definition in place, we are missing those three callback methods we are referencing in it. So, let's see the first one:

/**
 * Batch operation to remove the products which are no longer in the list of
 * products coming from the JSON file.
 *
 * @param $products
 * @param $context
 */
public function clearMissing($products, &$context) {
  if (!isset($context['results']['cleared'])) {
    $context['results']['cleared'] = [];
  }
  
  if (!$products) {
    return;
  }

  $ids = [];
  foreach ($products as $product) {
    $ids[] = $product->id;
  }

  $ids = $this->entityTypeManager->getStorage('product')->getQuery()
    ->condition('remote_id', $ids, 'NOT IN')
    ->execute();
  if (!$ids) {
    $context['results']['cleared'] = [];
    return;
  }

  $entities = $this->entityTypeManager->getStorage('product')->loadMultiple($ids);

  /** @var ProductInterface $entity */
  foreach ($entities as $entity) {
    $context['results']['cleared'][] = $entity->getName();
  }
  $context['message'] = $this->t('Removing @count products', ['@count' => count($entities)]);
  $this->entityTypeManager->getStorage('product')->delete($entities);
}

This is the first operation in the batch process. As an argument, it receives all the variables we defined in the batch definition (in our case, the products array). But it also gets a $context array variable passed by a reference, which we can use similar to how we used $sandbox in the update hook (with some extra capabilities).

The task at hand is pretty simple. We prepare a list of IDs of all the products in the JSON, and based on those, we query our product entities for the those which are NOT IN that list. If any are found, we delete them. You'll notice already that in this operation we are not relying on the actual multi-request capabilities of Drupal's Batch API because we expect the workload to be minimal. After all, how many products could be missing at any given time and would need to be deleted? We'll assume not many for our use case.

But while we are doing all this, we are interacting somewhat with the batch processing. You'll notice that the $context array has a results key. This is used to store information related to the outcome of each operation in the batch. We are not supposed to use it for managing progress but instead to keep track of what was done so that at the end we can present the user with some useful information as to what has happened. So in our example, we create an array keyed by cleared (to namespace the data for this particular operation), in which we put in the names of each product that has been deleted.

Moreover, we also have a message key that we use to print a message as the action is happening. This gets printed out in "real time" to indicate to the user what is currently being processed. If the batch is run via the UI through a form, it very well might be that you won't see all the messages due to the speed of the processing. However, if triggered by Drush (as it will be in our case), each of these messages will be printed to the terminal screen.

With this, our first operation is done. It's time to look at the second, more complex one:

/**
 * Batch operation to import the products from the JSON file.
 *
 * @param $products
 * @param $context
 */
public function importProducts($products, &$context) {
  if (!isset($context['results']['imported'])) {
    $context['results']['imported'] = [];
  }

  if (!$products) {
    return;
  }

  $sandbox = &$context['sandbox'];
  if (!$sandbox) {
    $sandbox['progress'] = 0;
    $sandbox['max'] = count($products);
    $sandbox['products'] = $products;
  }

  $slice = array_splice($sandbox['products'], 0, 3);
  foreach ($slice as $product) {
    $context['message'] = $this->t('Importing product @name', ['@name' => $product->name]);
    $this->persistProduct($product);
    $context['results']['imported'][] = $product->name;
    $sandbox['progress']++;
  }

  $context['finished'] = $sandbox['progress'] / $sandbox['max'];
}

The arguments it receives are exactly the same as with our previous operation since we defined them in the same way.

Here, again is we ensure we have some products and start up our results array, this time to keep track of the imported records. But we also work with the sandbox key of the $context array this time in order to use the multirequest processing capabilities. The approach is similar to what we did in the update hook--we keep a progress count, store the maximum number of products, and then we calculate the $context['finished'] key based on the division between the two. However, in this case, we opt to process three products at a time instead of one, as we did with our player records. Again, as with our previous operation, we are using the message key to inform the user as to what is going on and the results key to compile a list of products that have been imported.

Before moving on, let's talk a bit about the way we are importing the products. Had the JSON resource been able to return paginated results, we would have had to change our approach. First, we could not have deleted the missing products in the same way. Instead, we would have had to keep track of the IDs of the products that would have been imported only after deleting the other ones. So the order of two operations would have been reversed.

Second, the retrieval of the products would have been done from inside the importProducts operation using an offset and a limit stored in the sandbox. So, each Drupal batch request would have made a new request to the JSON resource. Of course, we would have had to keep track of all the processed products so that we would know which ones were able to be deleted.

Finally, let's take a look at the callback used when the batch processing finishes:

/**
 * Callback for when the batch processing completes.
 *
 * @param $success
 * @param $results
 * @param $operations
 */
public function importProductsFinished($success, $results, $operations) {
  if (!$success) {
    drupal_set_message($this->t('There was a problem with the batch'), 'error');
    return;
  }

  $cleared = count($results['cleared']);
  if ($cleared == 0) {
    drupal_set_message($this->t('No products had to be deleted.'));
  }
  else {
    drupal_set_message($this->formatPlural($cleared, '1 product had to be deleted.', '@count products had to be deleted.'));
  }

  $imported = count($results['imported']);
  if ($imported == 0) {
    drupal_set_message($this->t('No products found to be imported.'));
  }
  else {
    drupal_set_message($this->formatPlural($imported, '1 product imported.', '@count products imported.'));
  }
}

This callback receives three parameters--a boolean indicating whether the processing was successful or not, the results array we used inside our $context to keep track of what has been done, and the array of operations. What we are doing is actually pretty simple. We first print a generic message if the batch has failed. In this case, we also return early. Otherwise, we print relevant messages to the operations we have done, using the $results array. Note the use of the t() and formatPlural() methods you learned about in the previous chapter.

Or reworked JSON Importer now uses batching to make the process more stable in case the number of records it needs to process gets too big. Before we can try it out, we need to do one last step, and that is to use the DependencySerializationTrait inside the ImporterBase plugin class:

use DependencySerializationTrait;

The reason is that when the batch runs, Drupal stores some information about the object that runs it. In order to do so, it needs to serialize it. However, since it has dependencies such as the EntityTypeManager, Drupal needs a way to handle these in the serialization process. The preceding trait helps with that. Moreover, we can use it in the base class so that all plugin classes can use batching easily without having to worry about this step.

But now if we run the Drush command we wrote in Chapter 7, Your Own Custom Entity and Plugin Types, to trigger our importer, we get an output like this:

We notice the messages set when importing each record, as well as the messages we set at the end of the process, which provides kind of a summary of what went down.