None

WallpapersCraft.com Downloader


Automate the wallpaper download process from WallpapersCraft.com with php.

By Dimitris Dimitropoulos

There are a lot of websites offering wallpaper images and many of them also offer an API to automatically download them via an application interface. This script demonstrates how to parse the website HTML directly, while offering a simple public class to make image downloads as easy as possible.

The first part of the script sets the private variables. Some of these are specific to the WallpapersCraft.com website, like the categories. It is vital to understand how pages and the cache are being used by this script, in order to make changes and updates. Once the script receives the HTML output from the website, it is parsed for possible pagination results, based on the page limit, the script will request each page separately and add it to the page array. The results as stored by category in the cache, thus subsequent requests return hits from the cache array.

Be warned that our script contains a lot of debug “echo” lines, which are obviously suboptimal and should not be included in a usable script, but they are there to help users learn more about the way the script is working (or not).

 wcd.php

  1. #!/usr/bin/php
  2. <?php
  3. /*
  4.  * WallpapersCraft.com Downloader
  5.  */
  6.  
  7. Class WCDRecord {
  8.  
  9.     /* PRIVATE VARIABLES */
  10.  
  11.     // Debug mode
  12.     private $debug = false;
  13.  
  14.     // Category URL part
  15.     private $category = "";
  16.  
  17.     // Resolution URL part
  18.     private $resolution = "";
  19.  
  20.     // Sort by URL part
  21.     private $sortby = "";
  22.  
  23.     // List of category names and URL parts
  24.     private $categories = array(
  25.         '3D'        =>  "/3d",
  26.         'Abstract'  =>  "/abstract",
  27.         'Animals'   =>  "/animals",
  28.         'Anime'     =>  "/anime",
  29.         'Brands'    =>  "/brands",
  30.         'Cars'      =>  "/cars",
  31.         'City'      =>  "/city",
  32.         'Fantasy'   =>  "/fantasy",
  33.         'Flowers'   =>  "/flowers",
  34.         'Food'      =>  "/food",
  35.         'Games'     =>  "/games",
  36.         'Girls'     =>  "/girls",
  37.         'Hi-Tech'   =>  "/hi-tech",
  38.         'Holidays'  =>  "/holidays",
  39.         'Macro'     =>  "/macro",
  40.         'Men'       =>  "/men",
  41.         'Movies'    =>  "/movies",
  42.         'Music'     =>  "/music",
  43.         'Nature'    =>  "/nature",
  44.         'Other'     =>  "/other",
  45.         'Space'     =>  "/space",
  46.         'Sport'     =>  "/sport",
  47.         'Textures'  =>  "/textures",
  48.         'TV Series' =>  "/tv-series",
  49.         'Vector'    =>  "/vector"
  50.     );
  51.  
  52.     // List of sorting names and URL parts
  53.     private $sortings = array(
  54.         'Ratings'   =>  "",
  55.         'Downloads' =>  "/downloads",
  56.         'Date'      =>  "/date"
  57.     );
  58.  
  59.     // Main domain URL
  60.     private $mainUrl = 'https://wallpaperscraft.com';
  61.  
  62.     // User agent for curl calls
  63.     private $userAgent = 'Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0';
  64.  
  65.     // Cookie file
  66.     private $cookieFile = '/tmp/wcd-cookie.dat';
  67.  
  68.     // Temporary file
  69.     private $tmpFile = '/tmp/wcd-tmp.jpg';
  70.  
  71.     // Referer URL
  72.     private $referer = '';
  73.  
  74.     // HTML temporary cache (array, each element represents a different page of the same category/resolution/sort combination)
  75.     private $html = array();
  76.  
  77.     // Page limit
  78.     private $pageLimit = 20;
  79.  
  80.     // Page cache data file
  81.     private $cacheFile = '/tmp/.wcd-cache';
  82.  
  83.     // Page cache
  84.     private $pages = array();
  85.  
  86.     // Image cache
  87.     private $images = array();

 

Below are the private functions of the script. The getHTML() function uses CURL to request HTML pages from the website, the getFile() function uses CURL to request image files directly, the parsePages() function parses the HTML script for possible pagination results, the getPages() function is a wrapper function to download all the available pages based on the initial request of the script and the available cache array, the getImageList() function scans the downloaded pages for possible images to download and the selectRandomImage() function just picks one random image to download.

 wcd.php

  1. /* PRIVATE FUNCTIONS */
  2.  
  3.     private function getHTML($url) {
  4.  
  5.         // Cache
  6.         if(isset($this->pages[$url])) {
  7.             if($this->debug==true) echo "[WCD| Get page from cache ".$url."]\n";
  8.             return $this->pages[$url];
  9.         }
  10.  
  11.         // Get URL
  12.  
  13.         if($this->debug==true) echo "[WCD| Get page ".$url."]\n";
  14.  
  15.         // Initialize curl
  16.         $handle = curl_init();
  17.  
  18.         // Set curl options
  19.         curl_setopt($handle, CURLOPT_URL, $url);
  20.         curl_setopt($handle, CURLOPT_TIMEOUT, 30);
  21.         curl_setopt($handle, CURLOPT_USERAGENT, $this->userAgent);
  22.         curl_setopt($handle, CURLOPT_HEADER, false);
  23.         curl_setopt($handle, CURLOPT_RETURNTRANSFER, true);
  24.         curl_setopt($handle, CURLOPT_FOLLOWLOCATION, true);
  25.         curl_setopt($handle, CURLOPT_COOKIEFILE, $this->cookieFile);
  26.         curl_setopt($handle, CURLOPT_COOKIEJAR, $this->cookieFile);
  27.  
  28.         // Use a referer if one is available
  29.         if($this->referer!="")
  30.             curl_setopt($handle, CURLOPT_REFERER, $this->referer);
  31.  
  32.         if($this->debug==true) echo "[WCD| Curl execution...";
  33.  
  34.         // Execute curl
  35.         $rc = curl_exec($handle);
  36.  
  37.         if($this->debug==true) echo "OK]\n";
  38.  
  39.         // Handle curl error
  40.         if($rc===false && $this->debug==true) {
  41.             echo "[WCD| Curl returned error:\n";
  42.             echo curl_error($handle)."\n]\n";
  43.         }
  44.  
  45.         // Close curl connection
  46.         curl_close($handle);
  47.  
  48.         if($rc!==false) {
  49.             // Store URL for future use as a referer
  50.             $this->referer = $url;
  51.  
  52.             // Store page to cache
  53.             $this->pages[$url] = $rc;
  54.         }
  55.  
  56.         return $rc;
  57.  
  58.     }
  59.  
  60.     private function getFile($url, $dest) {
  61.  
  62.         // Get URL
  63.  
  64.         if($this->debug==true) echo "[WCD| Get file ".$url."]\n";
  65.  
  66.         // File pointer
  67.         $fp = fopen ($dest, 'w+');
  68.  
  69.         // Initialize curl
  70.         $handle = curl_init();
  71.  
  72.         // Set curl options
  73.         curl_setopt($handle, CURLOPT_URL, $url);
  74.         curl_setopt($handle, CURLOPT_TIMEOUT, 30);
  75.         curl_setopt($handle, CURLOPT_USERAGENT, $this->userAgent);
  76.         curl_setopt($handle, CURLOPT_HEADER, false);
  77.         curl_setopt($handle, CURLOPT_RETURNTRANSFER, true);
  78.         curl_setopt($handle, CURLOPT_FOLLOWLOCATION, true);
  79.         curl_setopt($handle, CURLOPT_COOKIEFILE, $this->cookieFile);
  80.         curl_setopt($handle, CURLOPT_COOKIEJAR, $this->cookieFile);
  81.         curl_setopt($handle, CURLOPT_FILE, $fp);
  82.  
  83.         // Use a referer if one is available
  84.         if($this->referer!="")
  85.             curl_setopt($handle, CURLOPT_REFERER, $this->referer);
  86.  
  87.         // Curl debug mode (disabled)
  88.         //if($this->debug==true) {
  89.         //  curl_setopt($handle, CURLOPT_VERBOSE, true);
  90.         //  curl_setopt($handle, CURLOPT_CERTINFO, true);
  91.         //}
  92.  
  93.         if($this->debug==true) echo "[WCD| Curl execution...";
  94.  
  95.         // Execute curl
  96.         $rc = curl_exec($handle);
  97.  
  98.         if($this->debug==true) echo "OK]\n";
  99.  
  100.         // Handle curl error
  101.         if($rc===false && $this->debug==true) {
  102.             echo "[WCD| Curl returned error:\n";
  103.             echo curl_error($handle)."\n]\n";
  104.         }
  105.  
  106.         // Close curl connection
  107.         curl_close($handle);
  108.  
  109.         // Close file pointer
  110.         fclose($fp);
  111.  
  112.         return $rc;
  113.  
  114.     }
  115.  
  116.     private function parsePages(&$html) {
  117.  
  118.         // Find start of pages div
  119.         $pos = mb_stripos($html, '<div class="pages">');
  120.         if($pos===false)
  121.             return false;
  122.  
  123.         // Find end of pages div
  124.         $pos2 = mb_stripos($html, '</div>', $pos);
  125.         if($pos2===false)
  126.             return false;
  127.  
  128.         // Extract pages div
  129.         $pages = mb_substr($html, $pos, $pos2-$pos);
  130.         if($pages===false || $pages=="")
  131.             return false;
  132.  
  133.         // Find last page
  134.         $pos = mb_strripos($pages, 'page_select');
  135.         if($pos===false)
  136.             return false;
  137.  
  138.         // Extract last page
  139.         $pages = mb_substr($pages, $pos);
  140.         $matches = array();
  141.         $rc = preg_match('/>(\d*)</ui', $pages, $matches);
  142.         if($rc!==1 || !isset($matches[1]))
  143.             return false;
  144.  
  145.         // Cast to integer
  146.         $matches[1] = (int)$matches[1];
  147.  
  148.         if($this->debug==true) echo "[WCD| Total number of pages: ".$matches[1].", limit set to: ".$this->pageLimit."]\n";
  149.  
  150.         return $matches[1];
  151.     }
  152.  
  153.     private function getPages() {
  154.  
  155.         // Generate complete URL
  156.         $url = $this->mainUrl.$this->category.$this->sortby.$this->resolution;
  157.  
  158.         if($this->debug==true) echo "[WCD| Page URL: ".$url."]\n";
  159.  
  160.         // Get 1st page
  161.         $this->html[0] = $this->getHTML($url);
  162.         if($this->html[0]===false) {
  163.             if($this->debug==true) echo "[WCD| Error while downloading HTML page from remote server]\n";
  164.             return false;
  165.         }
  166.  
  167.         // Parse total number of pages
  168.         $pages = $this->parsePages($this->html[0]);
  169.         if($pages===false) {
  170.             if($this->debug==true) echo "[WCD| Error while parsing for page numbers]\n";
  171.             return false;
  172.         }
  173.  
  174.         // Generate page array
  175.         $pageArray = array();
  176.         for($i=2;$i<=$pages;$i++)
  177.             $pageArray[$i] = $i;
  178.  
  179.         // Follow page limit, pick a random number of pages
  180.         if($pages > $this->pageLimit)
  181.             $pageArray = array_rand($pageArray, $this->pageLimit);
  182.  
  183.         // Loop and download all available pages
  184.         foreach($pageArray as $key => $i) {
  185.  
  186.             // Generate complete URL
  187.             $url = $this->mainUrl.$this->category.$this->sortby.$this->resolution.'/page'.$i;
  188.  
  189.             if($this->debug==true) echo "[WCD| Page URL: ".$url."]\n";
  190.  
  191.             // Get page
  192.             $rc = $this->getHTML($url);
  193.             if($rc===false) {
  194.                 if($this->debug==true) echo "[WCD| Error while downloading HTML page from remote server]\n";
  195.                 return false;
  196.             } else {
  197.                 $this->html[] = $rc;
  198.             }
  199.  
  200.         }
  201.  
  202.         // Save cache to file
  203.         if(file_exists($this->cacheFile)) {
  204.  
  205.             if($this->debug==true) echo "[WCD| Saving cache file: ".$this->cacheFile."]\n";
  206.  
  207.             // Store cache
  208.             $rc = file_put_contents($this->cacheFile, serialize($this->pages));
  209.             if($rc===false) {
  210.                 if($this->debug==true) echo "[WCD| Error while saving the cache file: ".$this->cacheFile."]\n";
  211.                 exit(9);
  212.             }
  213.  
  214.         }
  215.  
  216.     }
  217.  
  218.     private function getImageList() {
  219.  
  220.         foreach($this->html as $h) {
  221.  
  222.             // Find the start of the wallpapers div
  223.             $pos = mb_stripos($h, '<div class="wallpapers">');
  224.             if($pos===false)
  225.                 return false;
  226.  
  227.             // Find end of the wallpapers div
  228.             $pos2 = mb_stripos($h, '</div><div class="pages">', $pos);
  229.             if($pos2===false)
  230.                 return false;
  231.  
  232.             // Extract wallpapers div
  233.             $h = mb_substr($h, $pos, $pos2-$pos);
  234.             if($h===false || $h=="")
  235.                 return false;
  236.  
  237.             // Remove extra domain from URLs
  238.             $h = str_replace('//wallpaperscraft.com', '', $h);
  239.  
  240.             // Find all images
  241.             $matches = array();
  242.             $rc = preg_match_all('/<a href="(.*)" title="/ui', $h, $matches);
  243.             if($rc<1 || !isset($matches[1]))
  244.                 return false;
  245.  
  246.             // Store images to cache
  247.             $this->images = array_merge($this->images, $matches[1]);
  248.  
  249.         }
  250.  
  251.     }
  252.  
  253.     private function selectRandomImage() {
  254.  
  255.         // Return random image
  256.         return $this->images[array_rand($this->images, 1)];
  257.  
  258.     }

 

At this step, we define the public class structure. The WCDRecord() constructor initializes the cache, the function setCategory() selects the download image category, the setResolution() function sets the resolution requirements of the downloaded image file and the downloadImage() is the actual function that performs all the required actions to download an image file from the website.

The downloadImage() function also sets the downloaded image as the default desktop background, by default for GNOME3/Cinnamon/Mint background and optionally for GNOME2.

 wcd.php

  1.     /* PUBLIC FUNCTIONS */
  2.  
  3.     function WCDRecord($debug=false) {
  4.         $this->debug = $debug;
  5.  
  6.         // Initialize cache
  7.         if(file_exists($this->cacheFile)) {
  8.  
  9.             if($this->debug==true) echo "[WCD| Loading cache file: ".$this->cacheFile."]\n";
  10.  
  11.             // Load existing cache
  12.             $this->pages = @unserialize(file_get_contents($this->cacheFile));
  13.             if($this->pages===false || !is_array($this->pages)) {
  14.                 if($this->debug==true) echo "[WCD| Error while loading the cache file: ".$this->cacheFile."]\n";
  15.                 exit(9);
  16.             }
  17.  
  18.         } else {
  19.  
  20.             if($this->debug==true) echo "[WCD| Set empty cache file: ".$this->cacheFile."]\n";
  21.  
  22.             // Set empty cache
  23.             $this->pages = array();
  24.  
  25.             // Store empty cache for future use
  26.             $rc = file_put_contents($this->cacheFile, serialize($this->pages));
  27.             if($rc===false) {
  28.                 if($this->debug==true) echo "[WCD| Error while saving the cache file: ".$this->cacheFile."]\n";
  29.                 exit(9);
  30.             }
  31.  
  32.         }
  33.  
  34.     }
  35.  
  36.     function setCategory($category) {
  37.         if($category!=="" && !isset($this->categories[$category])) {
  38.             if($this->debug==true) echo "[WCD| Warning, requested category not recognized]\n";
  39.             $category = 'all';
  40.         }
  41.         if(($category==="" || $category==="all") && $this->resolution==="")
  42.             $this->category = '';
  43.         elseif(($category==="" || $category==="all") && $this->resolution!=="")
  44.             $this->category = '/all';
  45.         else
  46.             $this->category = '/catalog'.$this->categories[$category];
  47.         if($this->debug==true) echo "[WCD| Set category to: ".$this->category."]\n";
  48.     }
  49.  
  50.     function setResolution($resolution) {
  51.         if($resolution==="") {
  52.             $this->resolution = '';
  53.             if($this->category==="/all")
  54.                 $this->category = '';
  55.         } else {
  56.             $this->resolution = '/'.$resolution;
  57.             if($this->category==="")
  58.                 $this->category = '/all';
  59.         }
  60.         if($this->debug==true) echo "[WCD| Set resolution to: ".$this->resolution."]\n";
  61.     }
  62.  
  63.     function setSortBy($sortby) {
  64.         if(!isset($this->sortings[$sortby])) {
  65.             if($this->debug==true) echo "[WCD| Warning, requested sorting not recognized]\n";
  66.             $sortby = 'Ratings';
  67.         }
  68.         if($this->category==="/all")
  69.             $sortby = 'Ratings';
  70.         else
  71.             $this->sortby = $this->sortings[$sortby];
  72.         if($this->debug==true) echo "[WCD| Set sort by to: ".$this->sortby."]\n";
  73.     }
  74.  
  75.     function downloadImage($destination) {
  76.  
  77.         // Get pages
  78.         $rc = $this->getPages();
  79.         if($rc===false)
  80.             return false;
  81.  
  82.         // Get image list
  83.         $rc = $this->getImageList();
  84.         if($rc===false)
  85.             return false;
  86.  
  87.         // Get random image from image list
  88.         $imagePage = $this->selectRandomImage();
  89.  
  90.         if($this->debug==true) echo "[WCD| Selected image page: ".$imagePage."]\n";
  91.  
  92.         // Generate image URL
  93.         $imageUrl = $this->mainUrl.'/image/'.str_replace('/', '_', str_replace('/download/', '', $imagePage)).'.jpg';
  94.  
  95.         if($this->debug==true) echo "[WCD| Image URL: ".$imageUrl."]\n";
  96.  
  97.         // Set referer
  98.         $this->referer = $this->mainUrl.$imagePage;
  99.  
  100.         if($this->debug==true) echo "[WCD| Save image to: ".$destination."]\n";
  101.  
  102.         // Delete temporary file
  103.         if(file_exists($this->tmpFile)) {
  104.             if($this->debug==true) echo "[WCD| Temporary file already exists, deleting]\n";
  105.             @unlink($this->tmpFile);
  106.         }
  107.  
  108.         // Get file
  109.         $rc = $this->getFile($imageUrl, $this->tmpFile);
  110.         if($rc===false) {
  111.             if($this->debug==true) echo "[WCD| Error while downloading image file from remote server]\n";
  112.             return false;
  113.         }
  114.  
  115.         // Get file size
  116.         $size = filesize($this->tmpFile);
  117.         if($size===false || $size==0) {
  118.             if($this->debug==true) echo "[WCD| File size error]\n";
  119.             return false;
  120.         }
  121.         if($this->debug==true) echo "[WCD| Size of downloaded image file: ".$size." bytes]\n";
  122.  
  123.         if($this->debug==true) echo "[WCD| Mogrify image...";
  124.  
  125.         // ImageMagick resize and crop
  126.         `/usr/bin/mogrify -quality 100% -filter Lanczos -distort Resize 3440x1440^ -gravity center -extent 3440x1440 "$this->tmpFile"`;
  127.  
  128.         if($this->debug==true) echo "OK]\n";
  129.  
  130.         // Move temporary file to destination, override destination if it exists
  131.         $rc = rename($this->tmpFile, $destination);
  132.         if($rc===false) {
  133.             if($this->debug==true) echo "[WCD| Error while moving temporary file to destination]\n";
  134.             return false;
  135.         }
  136.  
  137.         // Set GNOME2 background
  138.         //`/usr/bin/gconftool-2 -t string -s /desktop/gnome/background/picture_filename "$destination"`;
  139.  
  140.         // Set GNOME3/Cinnamon/Mint background
  141.         `/usr/bin/gsettings set org.cinnamon.desktop.background picture-uri "file://$destination"`;
  142.  
  143.         if($this->debug==true) echo "[WCD| Image set as background]\n";
  144.  
  145.     }
  146.  
  147. }

 

In the last step we demonstrate how to execute the script and call the public functions. Once the WCDRecord() class is instantiated, we download a random image from any category, at a resolution of  3840x2160, by sorting the results by date and saving the final image as background.jpg in the current directory.

 wcd.php

  1. $wcd = new WCDRecord(false);
  2. $wcd->setCategory("all");
  3. $wcd->setResolution("3840x2160");
  4. $wcd->setSortBy("Date");
  5. $wcd->downloadImage('background.jpg');

 

So what have we achieved so far? We managed to create a quick parser for a single website, a simple method to poll results from multiple pages and a simple way of modifying the downloaded image via ImageMagick to our required resolution. Finally, we automatically set our modified image as a background wallpaper.

What is missing, is a more abstract way of downloading images from multiple websites, but that is left as an exercise to the reader.

 


Authors

Categories

Tags

linux (31)
Ansible (12)
automation (12)
IBM (11)
python (11)
networking (7)
bash (6)
cluster (6)
Django (6)
proxy (5)
database (5)
cloud (5)
lxc (4)
Websphere (4)
DB2 (4)
analytics (4)
monitoring (4)
SQL (4)
hosting (4)
virtualenv (4)
Redis (4)
Apache (4)
raspberry pi (3)
Microfocus (3)
availability (3)
e-mail (3)
portal (3)
container (3)
log files (3)
ipset (3)
php (3)
Nginx (3)
PostgreSQL (3)
Squid (2)
Corosync (2)
Pacemaker (2)
sles (2)
Logstash (2)
epilis (2)
certificates (2)
fail2ban (2)
firewalld (2)
Celery (2)
systemd (2)
HTTP (2)
openvpn / caching / xmitip / z/os / ubuntu / JCL / COBOL / DNS / nrpe / Nagios / load utility / static site / wiki / PXE / https / Dovecot / postfix / AWS / Watson / Channels / Elasticsearch / Kibana / windows / Prometheus /

View epilis's profile on LinkedIn Visit us on facebook X epilis rss feed: Latest articles