Recently our technical support team took part in resolving an intriguing case with an online store, hosted by us. Our customer had decided to use WordPress with WooCommerce for their online store.
At first sight they did not expect anything challenging or extraordinary from that combination. Besides the design and main features of the store, products are also a fundamental part of it. But it occurred that adding products to the store in the desired layout was trickier than usual. Our team took part in overcoming that challenge and together we stumbled upon a few more land mines which we managed to handle.
While analyzing and testing we gained a lot of knowledge about generating images in WP when using WooCommerce as well as techniques for importing a large number of products in WooCommerce.
So this experience has become a part of our list with interesting cases and valuable discoveries. We present to your attention the next exciting case study, which we hope everyone in a similar situation will benefit from.
The Challenge
Our customer was redesigning their online store cell.bg as they had chosen our highest hosting plan SuperHosting. For the store our customer had prepared products in an import ready format. The number of products was quite impressive – they amounted nearly 25.000.
However, attempts for importing the products always failed. The process was never completed as it was interrupted at different stages or points of time. To a certain extend this was not unusual – the number of products was considerable and lots of resources were required for such process. We first tried to help by increasing the resources. We reached a temporary increase of six times above the hosting plan parameters. The amount of memory in the account was also increased to 12 GB , but it was still not enough.
Furthermore, as a consequence of the import attempts, the number of inodes into the account also increased multiple times. We reached the unbelievable 1 million and 300 thousand files (compared to 270 thousand initially offered with the hosting plan).
The result up to then was a website without all the products uploaded, importing procedure taking hours with lots of recourses, over a million files in the account as they were even not all the available ones since there were also many unnecessary files. So that was quite a challenging start for a redesigned online store.
The Approach to Handling the Challenge
We definitely needed another approach to launching the website. Redesigning an online store is related to a number of business activities and a situation like this can have a negative impact on multiple levels. Many times we have said that our success depends on our customers’ success. So we changed our approach and figured out an action plan to be carefully implemented step by step.
The Plan Included:
- Finding the reason for the large number of generated files;
- Changing the website’s settings to prevent adding images which are not used on the website;
- Starting Clean;
- Importing the products stepwise.
1. Finding the reason for the large number of generated files – Images in WordPress and WooCommerce
By default for all files uploaded WordPress generates additional resized files depending on the settings specified in the admin panel. In some cases it is possible that over 15 resized copies are generated for a file.
But why? Various image sizes are used for different resolutions with which the website might be loaded. Most contemporary themes are responsive. For the website to look equally good cross-platform, the images are generated in the required sizes.
So the resized images are separate files and are not accessible from the WordPress admin panel. However, they still exist in the hosting account.
Under the “Files” menu in WordPress you can access only the original images. The real number of the copies is available in the File manager in cPanel or via FTP. Files of the image copies have the same name as they differ in size.
Here are the files generated during the importing procedure only for one image on the website:
31496_adonit-dash-3-stylus_1932768443-50×50.jpg
31496_adonit-dash-3-stylus_1932768443-100×100.jpg
31496_adonit-dash-3-stylus_1932768443-120×120.jpg
31496_adonit-dash-3-stylus_1932768443-145×145.jpg
31496_adonit-dash-3-stylus_1932768443-150×150.jpg
31496_adonit-dash-3-stylus_1932768443-300×300.jpg
31496_adonit-dash-3-stylus_1932768443-600×600.jpg
31496_adonit-dash-3-stylus_1932768443-604×270.jpg
31496_adonit-dash-3-stylus_1932768443-750×330.jpg
Here the first image is the original (_1932768443.jpg) as all of the 9 other images have been additionally generated during the import when the original image was uploaded to the website.
Each of the products has 4 images in total with different product views.
Expectation: 1 product = 4 images; 25.000 products = 100.000 imagesя
Reality: 1 product = 16 images; 25.000 products = 400.000 images
While testing we reached a result of over 1 million images before even all products had been imported.
In-depth analysis of the case with generating so many images was conducted by Kiril as it revealed the sources of generation.
The images on the website were generated by:
- WordPress – Image Settings (thumbnail, medium, large): “Settings” » “Files”.
This setting resulted in automatic generation of 1 thumbnail – 150х150.jpg. The thumbnail is used for build-in WordPress galleries. Medium and large size images are generated only when the default WordPress theme is used. In our case the website was using a different theme. To prevent copy generation, you can enter a zero (0) in the given field. - WooCommerce – Settings for Product Images: “Appearance” » “Customize” » “WooCommerce” » “Product Images“.
Products in the store may require separate images with different sizes as we discovered copies from the import with the 300×300.jpg and 600×600.jpg sizes. It is important to mention that if sizes are changed from the product image settings, after the changes are saved copies with the upgraded sizes will be automatically generated. - Theme – more image sizes are set up in the theme’s system files. Each WordPress theme might generate images with different sizes. You can check the theme’s code as you look for the following functions: add_image_size() and set_post_thumbnail_size(). The following image copies were generated for the used theme 50×50.jpg, 145×145.jpg, 604×270.jpg and 750×330.jpg.
- Plugins – even plugins might set their own preferences for image sizes. A copy with 120×120.jpg size was generated by the Woocommerce Product Gallery Slider plugin. The other 100×100.jpg copy was generated by the dynamic size settings. Plugins such as Slider Revolution which enable you to create galleries and sliders on the website also might have their own settings for image sizes. For example the thumbnail settings for this plugin can be modified from Slider Settings » Navigation (if thumbnail navigation is used for the slider).
Sometimes image sizes may not be directly found in the code, but dynamically generated according to the user settings in the admin panel. For example, setting up 145×145 size for thumbnails is not introduced to the code of Slider Revolution (e.g. 145, 145). Therefore, there is no way to discover them if you search the code.
2. Changing the website’s settings to prevent adding images which are not used on the website
We sent results from the analysis to our customer. For this online store the customer decided they needed 3 versions of each original image. This required us to change settings of the website, theme and code. So we could then take our next step – importing the products.
3. Starting Clean
Why did not we try changing anything in the current situation? It is nearly impossible to handle over a million files. Analyzing which files should be kept or removed meant that we had to check each and every file which is a waste of valuable time and resources. The fact that image copies are not accessible from the website’s administration makes the task impossible to solve.
4. Importing products into WooCommerce
After we already knew that all files we were uploading were the necessary ones, Gabriela successfully imported all 25 thousand products to the online store. The import was performed according to a specially designed action plan.
At the initial stage we prepared XML files with the products as each of them contained about 5.000-6.000 products. We used the WP All Import plugin which provides multiple options and the opportunity of importing products by using an URL or an XML/CSV file, even a Cron job. This plugin enables you to import products, purchase orders, posts and any other information in WordPress.
The plugin is very powerful and popular, it works very well and is often used by many customers. So at that moment the import type and data volume were fundamental. The first import of approximately 6.000 products took two hours, but was successfully completed. The import is an intensive operation since it requires checking the website’s database if the data is available. If the data is missing, the respective product is created; title, description and category are added together with prices and the rest of the attributes, then the image is downloaded from the remote server, it is resized and saved into the account. Even if a product import takes only a few seconds, the more products are added, the more time-consuming is the whole procedure.
The main difference we made is that we imported the products with separate XML files as each of them contained about 1.000 products. The customer assisted with file preparation.
When you upload products in WooCommerce for the first time and you do not simply update availability and prices you need to divide products into the smallest possible portions.
We chose this option so that we had smaller portions for import to be successfully completed, to easily control the import and to be able to see what goes wrong if this happens.
The hosting account yet again had 6 times more resources. However, good practices for importing a large number of products are as follows:
- Local product import and then uploading files and database into the hosting account
When importing products cannot be performed through the website or another application, available in the hosting account, you might first perform local import (on another instance) and then upload files and database to the hosting account. - Importing through a Cron job, again portioned ⭐️
When the import requires a lot of resources, you can use a Cron job. This way the process will run at the background step by step and will not prevent the website’s normal operation.
! It is bad practice to believe that a script will run smoothly from the start to the end. It is possible that something impedes the script, it might be stopped from server protection, an error code is possible to occur, etc. A good approach is to plan running it in a way that ensures your website’s proper operation even if the script stops.
The WP All Import plugin has an option for Cron job import which enables importing products or updating product details.
We imported at stages as each stage included 1.000 products. We regularly generated backups, in case something went wrong. The last import took a few hours since data had already reached quite a considerable amount.
The conclusion:
A WordPress website might generate numerous image versions and most of them are not used. However, those versions are not accessible through WordPress Dashboard » Files.
Setting up the number of images in WooCommerce is not done in the admin panel. The number of images depends on WordPress, the theme, plugins and WooCommerce.
When importing a large number of products you must find a way to control the process step by step, through a Cron job or locally on another instance, then only transfer files and database.
We would like to thank our customer for the close collaboration while handling this challenge. Products are now successfully imported and the store can use the hosting account’s parameters to a maximum extend.
Failure at importing and millions of files at first sight look like a hosting issue and unreasonable increase of the system parameters. It is a common truth that there is no notion such as “unlimited resources” and if you do not track them, the service cannot function properly and guarantee reliability of the file system in the hosting accounts. In this specific case we tried walking in our customer’s shoes. We knew that our customer needed help. Our technical support team is meant to be here for this reason and we are always happy to help by discovering the reason for increased number of inodes, to give advice, ideas and guidance. We are glad we could bring this case to a happy ending.
The Look Ahead
The online store is now working as it should. We wish for it to be very successful, to have lots of purchases and satisfied customers. The step to follow is upgrade to a suitable hosting plan in the long term – in this case Managed VPS which guarantees smooth operation at times of active promotions and advertising campaigns, respectively when the traffic is larger.
