Speed Up Your Website With WebP

dreamt up by webguru in Uncategorized | Comments Off on Speed Up Your Website With WebP

Speed Up Your Website With WebP

Speed Up Your Website With WebP

Suzanne Scacca



(This is a sponsored post.) Spend enough time running websites through PageSpeed Insights and you’ll notice that Google has a major beef with traditional image formats like JPG, PNG and even GIF. As well it should.

Even if you resize your images to the exact specifications of your website and run them through a compressor, they can still put a strain on performance and run up bandwidth usage. Worse, all of that image manipulation can compromise the resulting quality.

Considering how important images are to web design, this isn’t an element we can dispose of so easily nor can we afford to cut corners when it comes to optimizing them. So, what’s the solution?

Here’s what Google suggests:

PageSpeed Insights WebP tips

PageSpeed Insights demonstrates how much storage and bandwidth websites stand to save with WebP. (Source: PageSpeed Insights) (Large preview)

Years ago, Google aimed to put a stop to this problem by creating a next-gen image format called WebP. You can see in this screenshot from PageSpeed Insights that Google recommends using WebP and other next-gen formats to significantly reduce the size of your images while preserving their quality.

And if .75 seconds doesn’t seem like much to you (at least in this example), it could make a big difference in the lives of your visitors, the people who sit there wondering how long is too long to wait. Just one less second of loading could make a world of difference to your conversion rate.

But is WebP the best solution for this problem? Today, we’re going to examine:

What is WebP?

Google developed WebP back in 2010 after acquiring a company called On2 Technologies. On2 had worked on a number of video compression technologies, which ended up serving as the basis for Google’s new audiovisual format WebM and next-gen image format WebP.

Originally, WebP used lossy compression in an attempt to create smaller yet still high-quality images for the web.


If .75 seconds doesn’t seem like much to you, it could make a big difference in the lives of your visitors, the people who sit there wondering how long is too long to wait.

Lossy Compression For WebP

Lossy compression is a form of compression used to greatly reduce the file sizes of JPGs and GIFs. In order to make that happen, though, some of the data (pixels) from the file needs to be dropped out or “lost”. This, in turn, leads to some degradation of the quality of the image, though it’s not always noticeable.

WebP entered the picture with a much more efficient use of lossy compression (which I’ll explain below) and became the much-needed successor to JPG.

You can see a great demonstration of this difference as KeyCDN compares the difference in file sizes of a compressed JPG vs. WebP:

KeyCDN compares original file size against compressed JPG and WebP

KeyCDN shows how five images differ in size between the original, a compressed JPG and a WebP. (Source: KeyCDN) (Large preview)

Notice how significant a difference this is in terms of file size, even after the JPG has been compressed to a comparable quality. As Adrian James explains here, though, you have to be careful with WebP compression.

“Compression settings don’t match up one-to-one with JPEG. Don’t expect a 50%-quality JPEG to match a 50%-quality WebP. Quality drops pretty sharply on the WebP scale, so start at a high quality and work your way down.”

Considering how much more file sizes shrink with WebP compared to JPG, though, that shouldn’t be too much of a sticking point. It’s just something to think about if you’re considering pushing the limits of what WebP can do even further.

Now, as time passed, Google continued to develop WebP technology, eventually getting it to a point where it would support not just true-color web graphics, but also XMP metadata, color profiles, tiling, animation, and transparency.

Eventually, Google brought lossless compression to WebP, turning it into a viable contender for PNG, too.

Lossless Compression For WebP

Lossless compression does not degrade image quality the way lossy does. Instead, it achieves smaller file sizes by removing excess metadata from the backend of the file. This way, the quality of the image remains intact while reducing its size. That said, lossless compression can’t achieve the kinds of file sizes lossy compression can.

That was until WebP’s lossless compression came along.

You can see some beautiful examples of how WebP’s lossy and lossless compression stands up against PNG in Google’s WebP galleries:

Google WebP Gallery comparison against PNG

The Google WebP Galleries show how PNG images compare in quality and size to compressed WebPs. (Source: Google) (Large preview)

If there’s any degradation in the quality of the WebP images, it’s going to be barely noticeable to your visitors. The only thing they’re really going to notice is how quickly your site loads.

What Are The Advantages Of Using WebP?

It’s not enough to say that WebP is “better” than JPG and PNG. It’s important to understand the mechanics of how WebP works and why it’s so beneficial to use over other file formats as a result.

With traditional image formats, compression always results in a tradeoff.

JPG lossy compression leads to degradation of the clarity and fineness of an image. Once applied, it cannot be reversed.

WebP lossy compression, on the other hand, uses what’s known as prediction coding to more accurately adjust the pixels in an image. As Google explains, there are other factors at work, too:

“Block adaptive quantization makes a big difference, too. Filtering helps at mid/low bitrates. Boolean arithmetic encoding provides 5%-10% compression gains compared to Huffman encoding.”

On average, Google estimates that WebP lossy compression results in files that are between 25% and 34% smaller than JPGs of the same quality.

As for PNG lossless compression, it does work well in maintaining the quality of an image, but it doesn’t have as significant an impact on image size as its JPG counterpart. And certainly not when compared to WebP.

WebP handles this type of compression more efficiently and effectively. This is due to the variety of compression techniques used as well as entropy encoding applied to images. Again, Google explains how it works:

“The transforms applied to the image include spatial prediction of pixels, color space transform, using locally emerging palettes, packing multiple pixels into one pixel and alpha replacement.”

On average, Google estimates that WebP lossless compression results in files that are roughly 26% smaller than PNGs of the same quality.

That’s not all. WebP has the ability to do something that no other file formats can do. Designers can use WebP lossy encoding on RGB colors and lossless encoding on images with transparent backgrounds (alpha channel).

Animated images, otherwise served in GIF format, also benefit from WebP compression systems. There are a number of reasons for this:

GIF WebP
Compression Lossless Lossless + lossy
RBG Color Support 8-bit 24-bit
Alpha Channel Support 1-bit 8-bit

As a result of this powerful combo of lossless and lossy compression, animated videos can get down to much smaller sizes than their GIF counterparts.

Google estimates the average reduction to be about 64% of the original size of a GIF when using lossy compression and 19% when using lossless.


Needless to say, there’s nothing that can beat WebP when it comes to speed while maintaining image integrity.

Acceptance Of WebP Among Browsers, Devices And CMS

As you can imagine, when WebP was first released, it was only supported by Google’s browsers and devices. Over time, though, other platforms have begun to provide support for WebP images.

That said, WebP still doesn’t have universal support, which can cause problems for web designers who use this image format by default.

Let’s take a look at where you can expect full acceptance of your WebP images, where you won’t and then we’ll discuss what you can do to get around this hiccup.

As of writing this in 2019, Can I use… has accounted for the following platforms that support WebP:

‘Can I Use’ data on WebP support

‘Can I Use’ breaks down which browsers and versions of those browsers provide support for WebP. (Source: Can I use…) (Large preview)

The latest versions of the following platforms are supported:

  • Edge
  • Firefox
  • Chrome
  • Opera
  • Opera Mini
  • Android Browser
  • Opera Mobile
  • Chrome for Android
  • Firefox for Android
  • UC Browser for Android
  • Samsung Internet
  • QQ Browser
  • Baidu Browser

The platforms that continue to hold back support are:

  • Internet Explorer
  • Safari
  • iOS Safari
  • KaiOS Browser

It’s not just browsers that are on the fence about WebP. Image editing software and content management systems are, too.

ImageMagick, Pixelmator and GIMP all support WebP, for instance. Sketch enables users to export files as WebP. And for software that doesn’t natively support WebP, like Photoshop, users can usually install a plugin which will allow them to open and save files as WebP.

Content management systems are in a similar spot. Some have taken the lead in moving their users over to WebP, whether they uploaded their files in that format or not. Shopify and Wix are two site builders that automatically convert and serve images in WebP format.

Although there are other platforms that don’t natively support WebP, there are usually extensions or plugins you can use to upload WebP images or convert uploaded ones into this next-gen format.

WordPress is one of those platforms. Drupal is another popular CMS that provides users with WebP modules that add WebP support. Magento is yet another.

It’s pretty rare not to find some sort of add-on support for WebP. The only example that I’m aware of that doesn’t accept it is Squarespace.

Challenges Of Converting And Delivering WebP

Okay, so WebP doesn’t have 100% support on the web. Not yet anyway. That’s okay. For the most part, we have some sort of workaround in terms of adding support to the tools we use to design and build websites.

But what do we do about the browser piece? If our visitors show up on an iOS device, how do we make sure they’re still served an image if our default image is WebP?

First, you need to know how to convert images into WebP.

Last year, front end developer Jeremy Wagner wrote up a guide for Smashing Magazine on this very topic. In it, he covers how to convert to WebP using:

  • Sketch,
  • Photoshop,
  • The command line,
  • Bash,
  • Node.js,
  • gulp,
  • Grunt,
  • webpack.

Any of these options will help you convert your PNGs and JPGs into WebPs. Your image editing software will only get you halfway to your destination though.

It’ll handle the conversion, but it won’t help you modify your origin server so that it knows when to deliver WebPs and when to deliver a traditional image format to visitors.

Some of these methods let you dictate how your server delivers images based on the restraints of your visitors’ browsers. Still, it takes a bit of work to modify the origin servers to make this happen. If you’re not comfortable doing that or you don’t want to deal with it, KeyCDN has a solution.

The Solution: Simplify WebP Delivery With KeyCDN

KeyCDN understands how important it is to have a website that loads at lightning-fast speeds. It’s what KeyCDN is in the business to do. That’s why it’s no surprise that it’s developed a built-in WebP caching and image processing solution that helps developers more easily deliver the right file formats to visitors.

What Is WebP Caching?

Caching is an integral part of keeping any website running fast. And WebP caching is only going to make it better. Essentially, it’s a form of content negotiation that takes place in the HTTP header.

It works like this:

Someone visits a website that has KeyCDN’s WebP caching enabled. The visitor’s browser sends an accept HTTP header as part of the request to the server with a list of asset types it prefers. But rather than go to the origin server (at the web host), the request is processed by the edge server (at KeyCDN). The edge server reviews the list of acceptable file types and sends a content-type header in response.

Here’s an example of how that might look:

curl -I 'https://ip.keycdn.com/example.jpg' -H 'accept: image/webp'
HTTP/2 200
server: keycdn-engine
date: Thu, 06 Jun 2019 08:29:50 GMT
content-type: image/webp
content-length: 56734
last-modified: Tue, 14 May 2019 23:36:28 GMT
etag: "5cdb50fc-1040a"
expires: Thu, 13 Jun 2019 08:29:50 GMT
cache-control: max-age=604800
x-ip: 1
x-ip-info: osz=56734 odim=700x467 ofmt=webp
x-cache: HIT
x-shield: active
x-edge-location: chzh
access-control-allow-origin: *
accept-ranges: bytes

An example of a content-type request that KeyCDN sends to browsers that accept WebP. (Source: KeyCDN)

So, for Google Chrome visitors, the content-type: image/webp would automatically be accepted and the cached WebP assets would be delivered to the browser.

For Safari users, on the other hand, the request would go unaccepted. But that’s okay. Your CDN will know which file format to send instead. In the first line in the example above, you can see that the original image format is JPG, so that’s the version of the file that would be delivered.

As you can see, there’s no need to modify the origin server or prepare multiple versions of your files in order to account for WebP compatibility. KeyCDN WebP caching handles all of it.

How Do You Use KeyCDN WebP Caching?

There are two ways in which KeyCDN users can take advantage of the WebP caching feature.

Image Processing Through KeyCDN

The first requires nothing more than flipping a switch and turning on KeyCDN’s image processing. Once enabled, the accept request header will automatically load.

You can, of course, use the image processing service for more than just WebP caching. You can use it to adjust the size, crop, rotation, blur, and other physical attributes of your delivered images. But if you’re trying to simplify your image delivery system and simply want to speed things up with WebP, just enable the feature and let KeyCDN do the work.

WebP Caching Through Your Origin Server

Let’s say that you generated your own WebP image assets. You can still reap the benefits of KeyCDN’s WebP caching solution.

To do this, you’ll need to correctly generate your WebPs. Again, here’s a link to the guide that shows you how to do that.

It’s then up to you to configure your origin server so that it only delivers WebPs when accept: image/webp is present. KeyCDN provides some examples of how you’ll do this with Nginx:

# http config block
map $http_accept $webp_ext {
    default "";
    "~*webp" ".webp";
}

# server config block
location ~* ^(/path/to/your/images/.+).(png|jpg)$ {
    set $img_path $1;
    add_header Vary Accept;
    try_files $img_path$webp_ext $uri =404;
}

KeyCDN demonstrates how you can modify the origin server with Nginx to deliver your own cached WebP assets. (Source: KeyCDN)

And with Apache:

<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteCond %{HTTP_ACCEPT} image/webp
    RewriteCond %{DOCUMENT_ROOT}/$1.webp -f
    RewriteRule ^(path/to/your/images.+)\.(jpe?g|png)$ $1.webp [T=image/webp,E=accept:1]
</IfModule>

<IfModule mod_headers.c>
    Header append Vary Accept env=REDIRECT_accept
</IfModule>

AddType image/webp .webp

KeyCDN demonstrates how you can modify the origin server with Apache to deliver your own cached WebP assets. (Source: KeyCDN)

Obviously, this option gives you more control over managing your image formats and how they’re served to visitors. That said, if you’re new to using WebP, KeyCDN’s automated WebP caching and image processing is probably your best bet.

An Alternative For WordPress And Magento Designers

If you design websites in WordPress or Magento, KeyCDN has plugins you can use to add WebP support and caching.

For WordPress, you’ll use KeyCDN’s custom Cache Enabler along with Optimus.

Cache Enabler plugin from KeyCDN

The Cache Enabler plugin offers delivery support for WebPs in WordPress. (Source: Cache Enabler) (Large preview)

Cache Enabler checks to see if your images have a WebP version. If it exists and the visitor’s browser supports it, that’s what it will deliver in the cached file. If it doesn’t exist, then it’ll simply turn to the JPG, PNG or GIF that’s there.

Magento developers have a simplified workaround for converting and delivering WebP, too. First, you’ll need to install the Webp extension. Then, you’ll have to configure the WebP binaries on your server.

Wrapping Up

There’s a reason why Google went to the trouble of developing a new image format and why more and more browsers, design systems and content management systems are supporting it.

Images can cause a lot of problems for websites that have otherwise been built to be lean and mean. If they’re not uploaded at the right size, if they’re not compressed and if caching isn’t enabled, your images could be the reason that your website’s speed is driving visitors away.

But with WebP, your website is sure to load more quickly. What’s more, there doesn’t need to be a tradeoff between image quality (or quantity!) in order to gain that speed. WebP efficiently compresses files while preserving the integrity of the image content.

If you’re really struggling to increase the speed of your website then WebP should be the next tool you turn to for help.

Smashing Editorial
(ms, ra, il)

Source: Smashing Magazine, Speed Up Your Website With WebP

“Create Once, Publish Everywhere” With WordPress

dreamt up by webguru in Uncategorized | Comments Off on “Create Once, Publish Everywhere” With WordPress

“Create Once, Publish Everywhere” With WordPress

“Create Once, Publish Everywhere” With WordPress

Leonardo Losoviz



COPE is a strategy for reducing the amount of work needed to publish our content into different mediums, such as website, email, apps, and others. First pioneered by NPR, it accomplishes its goal by establishing a single source of truth for content which can be used for all of the different mediums.

Having content that works everywhere is not a trivial task since each medium will have its own requirements. For instance, whereas HTML is valid for printing content for the web, this language is not valid for an iOS/Android app. Similarly, we can add classes to our HTML for the web, but these must be converted to styles for email.

The solution to this conundrum is to separate form from content: The presentation and the meaning of the content must be decoupled, and only the meaning is used as the single source of truth. The presentation can then be added in another layer (specific to the selected medium).

For example, given the following piece of HTML code, the <p> is an HTML tag which applies mostly for the web, and attribute class="align-center" is presentation (placing an element “on the center” makes sense for a screen-based medium, but not for an audio-based one such as Amazon Alexa):

<p class="align-center">Hello world!</p>

Hence, this piece of content cannot be used as a single source of truth, and it must be converted into a format which separates the meaning from the presentation, such as the following piece of JSON code:

{
  content: "Hello world!",
  placement: "center",
  type: "paragraph"
}

This piece of code can be used as a single source of truth for content since from it we can recreate once again the HTML code to use for the web, and procure an appropriate format for other mediums.

Why WordPress

WordPress is ideal to implement the COPE strategy due of several reasons:

  • It is versatile.
    The WordPress database model does not define a fixed, rigid content model; on the contrary, it was created for versatility, enabling to create varied content models through the use of meta field, which allow the storing of additional pieces of data for four different entities: posts and custom post types, users, comments, and taxonomies (tags and categories).
  • It is powerful.
    WordPress shines as a CMS (Content Management System), and its plugin ecosystem enables to easily add new functionalities.
  • It is widespread.
    It is estimated that 1/3rd of websites run on WordPress. Then, a sizable amount of people working on the web know about and are able to use, i.e. WordPress. Not just developers but also bloggers, salesmen, marketing staff, and so on. Then, many different stakeholders, no matter their technical background, will be able to produce the content which acts as the single source of truth.
  • It is headless.
    Headlessness is the ability to decouple the content from the presentation layer, and it is a fundamental feature for implementing COPE (as to be able to feed data to dissimilar mediums).

    Since incorporating the WP REST API into core starting from version 4.7, and more markedly since the launch of Gutenberg in version 5.0 (for which plenty of REST API endpoints had to be implemented), WordPress can be considered a headless CMS, since most WordPress content can be accessed through a REST API by any application built on any stack.

    In addition, the recently-created WPGraphQL integrates WordPress and GraphQL, enabling to feed content from WordPress into any application using this increasingly popular API. Finally, my own project PoP has recently added an implementation of an API for WordPress which allows to export the WordPress data as either REST, GraphQL or PoP native formats.

  • It has Gutenberg, a block-based editor that greatly aids the implementation of COPE because it is based on the concept of blocks (as explained in the sections below).

Blobs Versus Blocks To Represent Information

A blob is a single unit of information stored all together in the database. For instance, writing the blog post below on a CMS that relies on blobs to store information will store the blog post content on a single database entry — containing that same content:

<p>Look at this wonderful tango:</p>
<figure>
  
  <figcaption>An exquisite tango performance</figcaption>
</figure>

As it can be appreciated, the important bits of information from this blog post (such as the content in the paragraph, and the URL, the dimensions and attributes of the Youtube video) are not easily accessible: If we want to retrieve any of them on their own, we need to parse the HTML code to extract them — which is far from an ideal solution.

Blocks act differently. By representing the information as a list of blocks, we can store the content in a more semantic and accessible way. Each block conveys its own content and its own properties which can depend on its type (e.g. is it perhaps a paragraph or a video?).

For example, the HTML code above could be represented as a list of blocks like this:

{
  [
    type: "paragraph",
    content: "Look at this wonderful tango:"
  ],
  [
    type: "embed",
    provider: "Youtube",
    url: "https://www.youtube.com/embed/sxm3Xyutc1s",
    width: 951,
    height: 535,
    frameborder: 0,
    allowfullscreen: true,
    allow: "accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture",
    caption: "An exquisite tango performance"
  ]  
}

Through this way of representing information, we can easily use any piece of data on its own, and adapt it for the specific medium where it must be displayed. For instance, if we want to extract all the videos from the blog post to show on a car entertainment system, we can simply iterate all blocks of information, select those with type="embed" and provider="Youtube", and extract the URL from them. Similarly, if we want to show the video on an Apple Watch, we need not care about the dimensions of the video, so we can ignore attributes width and height in a straightforward manner.

How Gutenberg Implements Blocks

Before WordPress version 5.0, WordPress used blobs to store post content in the database. Starting from version 5.0, WordPress ships with Gutenberg, a block-based editor, enabling the enhanced way to process content mentioned above, which represents a breakthrough towards the implementation of COPE. Unfortunately, Gutenberg has not been designed for this specific use case, and its representation of the information is different to the one just described for blocks, resulting in several inconveniences that we will need to deal with.

Let’s first have a glimpse on how the blog post described above is saved through Gutenberg:

<!-- wp:paragraph --> 
<p>Look at this wonderful tango:</p> 
<!-- /wp:paragraph -->  

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/embed/sxm3Xyutc1s","type":"rich","providerNameSlug":"embed-handler","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} --> 
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich is-provider-embed-handler wp-embed-aspect-16-9 wp-has-aspect-ratio">
  
https://www.youtube.com/embed/sxm3Xyutc1s
<figcaption>An exquisite tango performance</figcaption> </figure> <!-- /wp:core-embed/youtube -->

From this piece of code, we can make the following observations:

Blocks Are Saved All Together In The Same Database Entry

There are two blocks in the code above:

<!-- wp:paragraph -->
<p>Look at this wonderful tango:</p>
<!-- /wp:paragraph -->
<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/embed/sxm3Xyutc1s","type":"rich","providerNameSlug":"embed-handler","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} --> 
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich is-provider-embed-handler wp-embed-aspect-16-9 wp-has-aspect-ratio">
  
https://www.youtube.com/embed/sxm3Xyutc1s
<figcaption>An exquisite tango performance</figcaption> </figure> <!-- /wp:core-embed/youtube -->

With the exception of global (also called “reusable”) blocks, which have an entry of their own in the database and can be referenced directly through their IDs, all blocks are saved together in the blog post’s entry in table wp_posts.

Hence, to retrieve the information for a specific block, we will first need to parse the content and isolate all blocks from each other. Conveniently, WordPress provides function parse_blocks($content) to do just this. This function receives a string containing the blog post content (in HTML format), and returns a JSON object containing the data for all contained blocks.

Block Type And Attributes Are Conveyed Through HTML Comments

Each block is delimited with a starting tag <!-- wp:{block-type} {block-attributes-encoded-as-JSON} --> and an ending tag <!-- /wp:{block-type} --> which (being HTML comments) ensure that this information will not be visible when displaying it on a website. However, we can’t display the blog post directly on another medium, since the HTML comment may be visible, appearing as garbled content. This is not a big deal though, since after parsing the content through function parse_blocks($content), the HTML comments are removed and we can operate directly with the block data as a JSON object.

Blocks Contain HTML

The paragraph block has "<p>Look at this wonderful tango:</p>" as its content, instead of "Look at this wonderful tango:". Hence, it contains HTML code (tags <p> and </p>) which is not useful for other mediums, and as such must be removed, for instance through PHP function strip_tags($content).

When stripping tags, we can keep those HTML tags which explicitly convey semantic information, such as tags <strong> and <em> (instead of their counterparts <b> and <i> which apply only to a screen-based medium), and remove all other tags. This is because there is a great chance that semantic tags can be properly interpreted for other mediums too (e.g. Amazon Alexa can recognize tags <strong> and <em>, and change its voice and intonation accordingly when reading a piece of text). To do this, we invoke the strip_tags function with a 2nd parameter containing the allowed tags, and place it within a wrapping function for convenience:

function strip_html_tags($content) 
{
  return strip_tags($content, '<strong><em>');
}

The Video’s Caption Is Saved Within The HTML And Not As An Attribute

As can be seen in the Youtube video block, the caption "An exquisite tango performance" is stored inside the HTML code (enclosed by tag <figcaption />) but not inside the JSON-encoded attributes object. As a consequence, to extract the caption, we will need to parse the block content, for instance through a regular expression:

function extract_caption($content) 
{
  $matches = [];
  preg_match('/<figcaption>(.*?)</figcaption>/', $content, $matches);
  if ($caption = $matches[1]) {
    return strip_html_tags($caption);
  }
  return null;
}

This is a hurdle we must overcome in order to extract all metadata from a Gutenberg block. This happens on several blocks; since not all pieces of metadata are saved as attributes, we must then first identify which are these pieces of metadata, and then parse the HTML content to extract them on a block-by-block and piece-by-piece basis.

Concerning COPE, this represents a wasted chance to have a really optimal solution. It could be argued that the alternative option is not ideal either, since it would duplicate information, storing it both within the HTML and as an attribute, which violates the DRY (Don’t Repeat Yourself) principle. However, this violation does already take place: For instance, attribute className contains value "wp-embed-aspect-16-9 wp-has-aspect-ratio", which is printed inside the content too, under HTML attribute class.

Adding content through Gutenberg

Adding content through Gutenberg (Large preview)

Implementing COPE

Note: I have released this functionality, including all the code described below, as WordPress plugin Block Metadata. You’re welcome to install it and play with it so you can get a taste of the power of COPE. The source code is available in this GitHub repo.

Now that we know what the inner representation of a block looks like, let’s proceed to implement COPE through Gutenberg. The procedure will involve the following steps:

  1. Because function parse_blocks($content) returns a JSON object with nested levels, we must first simplify this structure.
  2. We iterate all blocks and, for each, identify their pieces of metadata and extract them, transforming them into a medium-agnostic format in the process. Which attributes are added to the response can vary depending on the block type.
  3. We finally make the data available through an API (REST/GraphQL/PoP).

Let’s implement these steps one by one.

1. Simplifying The Structure Of The JSON Object

The returned JSON object from function parse_blocks($content) has a nested architecture, in which the data for normal blocks appear at the first level, but the data for a referenced reusable block are missing (only data for the referencing block are added), and the data for nested blocks (which are added within other blocks) and for grouped blocks (where several blocks can be grouped together) appear under 1 or more sublevels. This architecture makes it difficult to process the block data from all blocks in the post content, since on one side some data are missing, and on the other we don’t know a priori under how many levels data are located. In addition, there is a block divider placed every pair of blocks, containing no content, which can be safely ignored.

For instance, the response obtained from a post containing a simple block, a global block, a nested block containing a simple block, and a group of simple blocks, in that order, is the following:

[
  // Simple block
  {
    "blockName": "core/image",
    "attrs": {
      "id": 70,
      "sizeSlug": "large"
    },
    "innerBlocks": [],
    "innerHTML": "n<figure class="wp-block-image size-large"><img src="http://localhost/wp-content/uploads/2017/12/sandwich-1024x614.jpg" alt="" class="wp-image-70"/><figcaption>This is a normal block</figcaption></figure>n",
    "innerContent": [
      "n<figure class="wp-block-image size-large"><img src="http://localhost/wp-content/uploads/2017/12/sandwich-1024x614.jpg" alt="" class="wp-image-70"/><figcaption>This is a normal block</figcaption></figure>n"
    ]
  },
  // Empty block divider
  {
    "blockName": null,
    "attrs": [],
    "innerBlocks": [],
    "innerHTML": "nn",
    "innerContent": [
      "nn"
    ]
  },
  // Reference to reusable block
  {
    "blockName": "core/block",
    "attrs": {
      "ref": 218
    },
    "innerBlocks": [],
    "innerHTML": "",
    "innerContent": []
  },
  // Empty block divider
  {
    "blockName": null,
    "attrs": [],
    "innerBlocks": [],
    "innerHTML": "nn",
    "innerContent": [
      "nn"
    ]
  },
  // Nested block
  {
    "blockName": "core/columns",
    "attrs": [],
    // Contained nested blocks
    "innerBlocks": [
      {
        "blockName": "core/column",
        "attrs": [],
        // Contained nested blocks
        "innerBlocks": [
          {
            "blockName": "core/image",
            "attrs": {
              "id": 69,
              "sizeSlug": "large"
            },
            "innerBlocks": [],
            "innerHTML": "n<figure class="wp-block-image size-large"><img src="http://localhost/wp-content/uploads/2017/12/espresso-1024x614.jpg" alt="" class="wp-image-69"/></figure>n",
            "innerContent": [
              "n<figure class="wp-block-image size-large"><img src="http://localhost/wp-content/uploads/2017/12/espresso-1024x614.jpg" alt="" class="wp-image-69"/></figure>n"
            ]
          }
        ],
        "innerHTML": "n
n", "innerContent": [ "n
", null, "
n" ] }, { "blockName": "core/column", "attrs": [], // Contained nested blocks "innerBlocks": [ { "blockName": "core/paragraph", "attrs": [], "innerBlocks": [], "innerHTML": "n<p>This is how I wake up every morning</p>n", "innerContent": [ "n<p>This is how I wake up every morning</p>n" ] } ], "innerHTML": "n
n", "innerContent": [ "n
", null, "
n" ] } ], "innerHTML": "n
nn
n", "innerContent": [ "n
", null, "nn", null, "
n" ] }, // Empty block divider { "blockName": null, "attrs": [], "innerBlocks": [], "innerHTML": "nn", "innerContent": [ "nn" ] }, // Block group { "blockName": "core/group", "attrs": [], // Contained grouped blocks "innerBlocks": [ { "blockName": "core/image", "attrs": { "id": 71, "sizeSlug": "large" }, "innerBlocks": [], "innerHTML": "n<figure class="wp-block-image size-large"><img src="http://localhost/wp-content/uploads/2017/12/coffee-1024x614.jpg" alt="" class="wp-image-71"/><figcaption>First element of the group</figcaption></figure>n", "innerContent": [ "n<figure class="wp-block-image size-large"><img src="http://localhost/wp-content/uploads/2017/12/coffee-1024x614.jpg" alt="" class="wp-image-71"/><figcaption>First element of the group</figcaption></figure>n" ] }, { "blockName": "core/paragraph", "attrs": [], "innerBlocks": [], "innerHTML": "n<p>Second element of the group</p>n", "innerContent": [ "n<p>Second element of the group</p>n" ] } ], "innerHTML": "n
nn
n", "innerContent": [ "n
", null, "nn", null, "
n" ] } ]

A better solution is to have all data at the first level, so the logic to iterate through all block data is greatly simplified. Hence, we must fetch the data for these reusable/nested/grouped blocks, and have it added on the first level too. As it can be seen in the JSON code above:

  • The empty divider block has attribute "blockName" with value NULL
  • The reference to a reusable block is defined through $block["attrs"]["ref"]
  • Nested and group blocks define their contained blocks under $block["innerBlocks"]

Hence, the following PHP code removes the empty divider blocks, identifies the reusable/nested/grouped blocks and adds their data to the first level, and removes all data from all sublevels:

/**
 * Export all (Gutenberg) blocks' data from a WordPress post
 */
function get_block_data($content, $remove_divider_block = true)
{
  // Parse the blocks, and convert them into a single-level array
  $ret = [];
  $blocks = parse_blocks($content);
  recursively_add_blocks($ret, $blocks);

  // Maybe remove blocks without name
  if ($remove_divider_block) {
    $ret = remove_blocks_without_name($ret);
  }

  // Remove 'innerBlocks' property if it exists (since that code was copied to the first level, it is currently duplicated)
  foreach ($ret as &$block) {
    unset($block['innerBlocks']);
  }

  return $ret;
}

/**
 * Remove the blocks without name, such as the empty block divider
 */
function remove_blocks_without_name($blocks)
{
  return array_values(array_filter(
    $blocks,
    function($block) {
      return $block['blockName'];
    }
  ));
}

/**
 * Add block data (including global and nested blocks) into the first level of the array
 */
function recursively_add_blocks(&$ret, $blocks) 
{  
  foreach ($blocks as $block) {
    // Global block: add the referenced block instead of this one
    if ($block['attrs']['ref']) {
      $ret = array_merge(
        $ret,
        recursively_render_block_core_block($block['attrs'])
      );
    }
    // Normal block: add it directly
    else {
      $ret[] = $block;
    }
    // If it contains nested or grouped blocks, add them too
    if ($block['innerBlocks']) {
      recursively_add_blocks($ret, $block['innerBlocks']);
    }
  }
}

/**
 * Function based on `render_block_core_block`
 */
function recursively_render_block_core_block($attributes) 
{
  if (empty($attributes['ref'])) {
    return [];
  }

  $reusable_block = get_post($attributes['ref']);
  if (!$reusable_block || 'wp_block' !== $reusable_block->post_type) {
    return [];
  }

  if ('publish' !== $reusable_block->post_status || ! empty($reusable_block->post_password)) {
    return [];
  }

  return get_block_data($reusable_block->post_content);
}

Calling function get_block_data($content) passing the post content ($post->post_content) as parameter, we now obtain the following response:

[[
  {
    "blockName": "core/image",
    "attrs": {
      "id": 70,
      "sizeSlug": "large"
    },
    "innerHTML": "n<figure class="wp-block-image size-large"><img src="http://localhost/wp-content/uploads/2017/12/sandwich-1024x614.jpg" alt="" class="wp-image-70"/><figcaption>This is a normal block</figcaption></figure>n",
    "innerContent": [
      "n<figure class="wp-block-image size-large"><img src="http://localhost/wp-content/uploads/2017/12/sandwich-1024x614.jpg" alt="" class="wp-image-70"/><figcaption>This is a normal block</figcaption></figure>n"
    ]
  },
  {
    "blockName": "core/paragraph",
    "attrs": [],
    "innerHTML": "n<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.</p>n",
    "innerContent": [
      "n<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.</p>n"
    ]
  },
  {
    "blockName": "core/columns",
    "attrs": [],
    "innerHTML": "n
nn
n", "innerContent": [ "n
", null, "nn", null, "
n" ] }, { "blockName": "core/column", "attrs": [], "innerHTML": "n
n", "innerContent": [ "n
", null, "
n" ] }, { "blockName": "core/image", "attrs": { "id": 69, "sizeSlug": "large" }, "innerHTML": "n<figure class="wp-block-image size-large"><img src="http://localhost/wp-content/uploads/2017/12/espresso-1024x614.jpg" alt="" class="wp-image-69"/></figure>n", "innerContent": [ "n<figure class="wp-block-image size-large"><img src="http://localhost/wp-content/uploads/2017/12/espresso-1024x614.jpg" alt="" class="wp-image-69"/></figure>n" ] }, { "blockName": "core/column", "attrs": [], "innerHTML": "n
n", "innerContent": [ "n
", null, "
n" ] }, { "blockName": "core/paragraph", "attrs": [], "innerHTML": "n<p>This is how I wake up every morning</p>n", "innerContent": [ "n<p>This is how I wake up every morning</p>n" ] }, { "blockName": "core/group", "attrs": [], "innerHTML": "n
nn
n", "innerContent": [ "n
", null, "nn", null, "
n" ] }, { "blockName": "core/image", "attrs": { "id": 71, "sizeSlug": "large" }, "innerHTML": "n<figure class="wp-block-image size-large"><img src="http://localhost/wp-content/uploads/2017/12/coffee-1024x614.jpg" alt="" class="wp-image-71"/><figcaption>First element of the group</figcaption></figure>n", "innerContent": [ "n<figure class="wp-block-image size-large"><img src="http://localhost/wp-content/uploads/2017/12/coffee-1024x614.jpg" alt="" class="wp-image-71"/><figcaption>First element of the group</figcaption></figure>n" ] }, { "blockName": "core/paragraph", "attrs": [], "innerHTML": "n<p>Second element of the group</p>n", "innerContent": [ "n<p>Second element of the group</p>n" ] } ]

Even though not strictly necessary, it is very helpful to create a REST API endpoint to output the result of our new function get_block_data($content), which will allow us to easily understand what blocks are contained in a specific post, and how they are structured. The code below adds such endpoint under /wp-json/block-metadata/v1/data/{POST_ID}:

/**
 * Define REST endpoint to visualize a post’s block data
 */
add_action('rest_api_init', function () {
  register_rest_route('block-metadata/v1', 'data/(?Pd+)', [
    'methods'  => 'GET',
    'callback' => 'get_post_blocks'
  ]);
});
function get_post_blocks($request) 
{
  $post = get_post($request['post_id']);
  if (!$post) {
    return new WP_Error('empty_post', 'There is no post with this ID', array('status' => 404));
  }

  $block_data = get_block_data($post->post_content);
  $response = new WP_REST_Response($block_data);
  $response->set_status(200);
  return $response;
}

To see it in action, check out this link which exports the data for this post.

2. Extracting All Block Metadata Into A Medium-Agnostic Format

At this stage, we have block data containing HTML code which is not appropriate for COPE. Hence, we must strip the non-semantic HTML tags for each block as to convert it into a medium-agnostic format.

We can decide which are the attributes that must be extracted on a block type by block type basis (for instance, extract the text alignment property for "paragraph" blocks, the video URL property for the "youtube embed" block, and so on).

As we saw earlier on, not all attributes are actually saved as block attributes but within the block’s inner content, hence, for these situations, we will need to parse the HTML content using regular expressions in order to extract those pieces of metadata.

After inspecting all blocks shipped through WordPress core, I decided not to extract metadata for the following ones:

"core/columns"
"core/column"
"core/cover"
These apply only to screen-based mediums and (being nested blocks) are difficult to deal with.
"core/html" This one only makes sense for web.
"core/table"
"core/button"
"core/media-text"
I had no clue how to represent their data on a medium-agnostic fashion or if it even makes sense.

This leaves me with the following blocks, for which I’ll proceed to extract their metadata:

To extract the metadata, we create function get_block_metadata($block_data) which receives an array with the block data for each block (i.e. the output from our previously-implemented function get_block_data) and, depending on the block type (provided under property "blockName"), decides what attributes are required and how to extract them:

/**
 * Process all (Gutenberg) blocks' metadata into a medium-agnostic format from a WordPress post
 */
function get_block_metadata($block_data)
{
  $ret = [];
  foreach ($block_data as $block) {
    $blockMeta = null;
    switch ($block['blockName']) {
      case ...:
        $blockMeta = ...
        break;
      case ...:
        $blockMeta = ...
        break;
      ...
    }

    if ($blockMeta) {
      $ret[] = [
        'blockName' => $block['blockName'],
        'meta' => $blockMeta,
      ];
    }
  }

  return $ret;
}

Let’s proceed to extract the metadata for each block type, one by one:

"core/paragraph"

Simply remove the HTML tags from the content, and remove the trailing breaklines.

case 'core/paragraph':
  $blockMeta = [
    'content' => trim(strip_html_tags($block['innerHTML'])),
  ];
  break;

'core/image'

The block either has an ID referring to an uploaded media file or, if not, the image source must be extracted from under <img src="...">. Several attributes (caption, linkDestination, link, alignment) are optional.

case 'core/image':
  $blockMeta = [];
  // If inserting the image from the Media Manager, it has an ID
  if ($block['attrs']['id'] && $img = wp_get_attachment_image_src($block['attrs']['id'], $block['attrs']['sizeSlug'])) {
    $blockMeta['img'] = [
      'src' => $img[0],
      'width' => $img[1],
      'height' => $img[2],
    ];
  }
  elseif ($src = extract_image_src($block['innerHTML'])) {
    $blockMeta['src'] = $src;
  }
  if ($caption = extract_caption($block['innerHTML'])) {
    $blockMeta['caption'] = $caption;
  }
  if ($linkDestination = $block['attrs']['linkDestination']) {
    $blockMeta['linkDestination'] = $linkDestination;
    if ($link = extract_link($block['innerHTML'])) {
      $blockMeta['link'] = $link;
    }
  }
  if ($align = $block['attrs']['align']) {
    $blockMeta['align'] = $align;
  }
  break;

It makes sense to create functions extract_image_src, extract_caption and extract_link since their regular expressions will be used time and again for several blocks. Please notice that a caption in Gutenberg can contain links (<a href="...">), however, when calling strip_html_tags, these are removed from the caption.

Even though regrettable, I find this practice unavoidable, since we can’t guarantee a link to work in non-web platforms. Hence, even though the content is gaining universality since it can be used for different mediums, it is also losing specificity, so its quality is poorer compared to content that was created and customized for the particular platform.

function extract_caption($innerHTML)
{
  $matches = [];
  preg_match('/<figcaption>(.*?)</figcaption>/', $innerHTML, $matches);
  if ($caption = $matches[1]) {
    return strip_html_tags($caption);
  }
  return null;
}

function extract_link($innerHTML)
{
  $matches = [];
  preg_match('/<a href="(.*?)">(.*?)</a>>', $innerHTML, $matches);
  if ($link = $matches[1]) {
    return $link;
  }
  return null;
}

function extract_image_src($innerHTML)
{
  $matches = [];
  preg_match('/<img src="(.*?)"/', $innerHTML, $matches);
  if ($src = $matches[1]) {
    return $src;
  }
  return null;
}

'core-embed/youtube'

Simply retrieve the video URL from the block attributes, and extract its caption from the HTML content, if it exists.

case 'core-embed/youtube':
  $blockMeta = [
    'url' => $block['attrs']['url'],
  ];
  if ($caption = extract_caption($block['innerHTML'])) {
    $blockMeta['caption'] = $caption;
  }
  break;

'core/heading'

Both the header size (h1, h2, …, h6) and the heading text are not attributes, so these must be obtained from the HTML content. Please notice that, instead of returning the HTML tag for the header, the size attribute is simply an equivalent representation, which is more agnostic and makes better sense for non-web platforms.

case 'core/heading':
  $matches = [];
  preg_match('/<h[1-6])>(.*?)</h([1-6])>/', $block['innerHTML'], $matches);
  $sizes = [
    null,
    'xxl',
    'xl',
    'l',
    'm',
    'sm',
    'xs',
  ];
  $blockMeta = [
    'size' => $sizes[$matches[1]],
    'heading' => $matches[2],
  ];
  break;

Unfortunately, for the image gallery I have been unable to extract the captions from each image, since these are not attributes, and extracting them through a simple regular expression can fail: If there is a caption for the first and third elements, but none for the second one, then I wouldn’t know which caption corresponds to which image (and I haven’t devoted the time to create a complex regex). Likewise, in the logic below I’m always retrieving the "full" image size, however, this doesn’t have to be the case, and I’m unaware of how the more appropriate size can be inferred.

case 'core/gallery':
  $imgs = [];
  foreach ($block['attrs']['ids'] as $img_id) {
    $img = wp_get_attachment_image_src($img_id, 'full');
    $imgs[] = [
      'src' => $img[0],
      'width' => $img[1],
      'height' => $img[2],
    ];
  }
  $blockMeta = [
    'imgs' => $imgs,
  ];
  break;

'core/list'

Simply transform the <li> elements into an array of items.

case 'core/list':
  $matches = [];
  preg_match_all('/<li>(.*?)</li>/', $block['innerHTML'], $matches);
  if ($items = $matches[1]) {
    $blockMeta = [
      'items' => array_map('strip_html_tags', $items),
    ];
  }
  break;

'core/audio'

Obtain the URL of the corresponding uploaded media file.

case 'core/audio':
  $blockMeta = [
    'src' => wp_get_attachment_url($block['attrs']['id']),
  ];
  break;

'core/file'

Whereas the URL of the file is an attribute, its text must be extracted from the inner content.

case 'core/file':
  $href = $block['attrs']['href'];
  $matches = [];
  preg_match('/<a href="'.str_replace('/', '/', $href).'">(.*?)</a>/', $block['innerHTML'], $matches);
  $blockMeta = [
    'href' => $href,
    'text' => strip_html_tags($matches[1]),
  ];
  break;

'core/video'

Obtain the video URL and all properties to configure how the video is played through a regular expression. If Gutenberg ever changes the order in which these properties are printed in the code, then this regex will stop working, evidencing one of the problems of not adding metadata directly through the block attributes.

case 'core/video':
  $matches = [];
  preg_match('/

'core/code'

Simply extract the code from within <code />.

case 'core/code':
  $matches = [];
  preg_match('/<code>(.*?)</code>/is', $block['innerHTML'], $matches);
  $blockMeta = [
    'code' => $matches[1],
  ];
  break;

'core/preformatted'

Similar to <code />, but we must watch out that Gutenberg hardcodes a class too.

case 'core/preformatted':
  $matches = [];
  preg_match('/<pre class="wp-block-preformatted">(.*?)</pre>/is', $block['innerHTML'], $matches);
  $blockMeta = [
    'text' => strip_html_tags($matches[1]),
  ];
  break;

'core/quote' and 'core/pullquote'

We must convert all inner <p /> tags to their equivalent generic "n" character.

case 'core/quote':
case 'core/pullquote':
  $matches = [];
  $regexes = [
    'core/quote' => '/<blockquote class="wp-block-quote">(.*?)</blockquote>/',
    'core/pullquote' => '/<figure class="wp-block-pullquote"><blockquote>(.*?)</blockquote></figure>/',
  ];
  preg_match($regexes[$block['blockName']], $block['innerHTML'], $matches);
  if ($quoteHTML = $matches[1]) {
    preg_match_all('/<p>(.*?)</p>/', $quoteHTML, $matches);
    $blockMeta = [
      'quote' => strip_html_tags(implode('n', $matches[1])),
    ];
    preg_match('/<cite>(.*?)</cite>/', $quoteHTML, $matches);
    if ($cite = $matches[1]) {
      $blockMeta['cite'] = strip_html_tags($cite);
    }
  }
  break;

'core/verse'

Similar situation to <pre />.

case 'core/verse':
  $matches = [];
  preg_match('/<pre class="wp-block-verse">(.*?)</pre>/is', $block['innerHTML'], $matches);
  $blockMeta = [
    'text' => strip_html_tags($matches[1]),
  ];
  break;

3. Exporting Data Through An API

Now that we have extracted all block metadata, we need to make it available to our different mediums, through an API. WordPress has access to the following APIs:

Let’s see how to export the data through each of them.

REST

The following code creates endpoint /wp-json/block-metadata/v1/metadata/{POST_ID} which exports all block metadata for a specific post:

/**
 * Define REST endpoints to export the blocks' metadata for a specific post
 */
add_action('rest_api_init', function () {
  register_rest_route('block-metadata/v1', 'metadata/(?Pd+)', [
    'methods'  => 'GET',
    'callback' => 'get_post_block_meta'
  ]);
});
function get_post_block_meta($request) 
{
  $post = get_post($request['post_id']);
  if (!$post) {
    return new WP_Error('empty_post', 'There is no post with this ID', array('status' => 404));
  }

  $block_data = get_block_data($post->post_content);
  $block_metadata = get_block_metadata($block_data);
  $response = new WP_REST_Response($block_metadata);
  $response->set_status(200);
  return $response;
}

To see it working, this link (corresponding to this blog post) displays the metadata for blocks of all the types analyzed earlier on.

GraphQL (Through WPGraphQL)

GraphQL works by setting-up schemas and types which define the structure of the content, from which arises this API’s power to fetch exactly the required data and nothing else. Setting-up schemas works very well when the structure of the object has a unique representation.

In our case, however, the metadata returned by a new field "block_metadata" (which calls our newly-created function get_block_metadata) depends on the specific block type, so the structure of the response can vary wildly; GraphQL provides a solution to this issue through a Union type, allowing to return one among a set of different types. However, its implementation for all different variations of the metadata structure has proved to be a lot of work, and I quit along the way 😢.

As an alternative (not ideal) solution, I decided to provide the response by simply encoding the JSON object through a new field "jsonencoded_block_metadata":

/**
 * Define WPGraphQL field "jsonencoded_block_metadata"
 */
add_action('graphql_register_types', function() {
  register_graphql_field(
    'Post',
    'jsonencoded_block_metadata',
    [
      'type'        => 'String',
      'description' => __('Post blocks encoded as JSON', 'wp-graphql'),
      'resolve'     => function($post) {
        $post = get_post($post->ID);
        $block_data = get_block_data($post->post_content);
        $block_metadata = get_block_metadata($block_data);
        return json_encode($block_metadata);
      }
    ]
  );
});

PoP

Note: This functionality is available on its own GitHub repo.

The final API is called PoP, which is a little-known project I’ve been working on for several years now. I have recently converted it into a full-fledged API, with the capacity to produce a response compatible with both REST and GraphQL, and which even benefits from the advantages from these 2 APIs, at the same time: no under/over-fetching of data, like in GraphQL, while being cacheable on the server-side and not susceptible to DoS attacks, like REST. It offers a mix between the two of them: REST-like endpoints with GraphQL-like queries.

The block metadata is made available through the API through the following code:

class PostFieldValueResolver extends AbstractDBDataFieldValueResolver
{
  public static function getClassesToAttachTo(): array
  {
    return array(PoPPostsFieldResolver::class);
  }

  public function resolveValue(FieldResolverInterface $fieldResolver, $resultItem, string $fieldName, array $fieldArgs = [])
  {
    $post = $resultItem;
    switch ($fieldName) {
      case 'block-metadata':
        $block_data = LeolosoBlockMetadataData::get_block_data($post->post_content);
        $block_metadata = LeolosoBlockMetadataMetadata::get_block_metadata($block_data);

        // Filter by blockName
        if ($blockName = $fieldArgs['blockname']) {
          $block_metadata = array_filter(
            $block_metadata,
            function($block) use($blockName) {
              return $block['blockName'] == $blockName;
            }
          );
        }
        return $block_metadata;
    }

    return parent::resolveValue($fieldResolver, $resultItem, $fieldName, $fieldArgs);
  }
}

To see it in action, this link displays the block metadata (+ ID, title and URL of the post, and the ID and name of its author, à la GraphQL) for a list of posts.

In addition, similar to GraphQL arguments, our query can be customized through field arguments, enabling to obtain only the data that makes sense for a specific platform. For instance, if we desire to extract all Youtube videos added to all posts, we can add modifier (blockname:core-embed/youtube) to field block-metadata in the endpoint URL, like in this link. Or if we want to extract all images from a specific post, we can add modifier (blockname:core/image) like in this other link|id|title).

Conclusion

The COPE (“Create Once, Publish Everywhere”) strategy helps us lower the amount of work needed to create several applications which must run on different mediums (web, email, apps, home assistants, virtual reality, etc) by creating a single source of truth for our content. Concerning WordPress, even though it has always shined as a Content Management System, implementing the COPE strategy has historically proved to be a challenge.

However, a couple of recent developments have made it increasingly feasible to implement this strategy for WordPress. On one side, since the integration into core of the WP REST API, and more markedly since the launch of Gutenberg, most WordPress content is accessible through APIs, making it a genuine headless system. On the other side, Gutenberg (which is the new default content editor) is block-based, making all metadata inside a blog post readily accessible to the APIs.

As a consequence, implementing COPE for WordPress is straightforward. In this article, we have seen how to do it, and all the relevant code has been made available through several repositories. Even though the solution is not optimal (since it involves plenty of parsing HTML code), it still works fairly well, with the consequence that the effort needed to release our applications to multiple platforms can be greatly reduced. Kudos to that!

Smashing Editorial
(dm, il)

Source: Smashing Magazine, “Create Once, Publish Everywhere” With WordPress

Collective #561

dreamt up by webguru in Uncategorized | Comments Off on Collective #561







C561_overview

Overview

A beautiful project: Overview uses satellite and aerial imagery to demonstrate how human activity and natural forces shape our Earth.

Check it out















C561_todo

CLARO

A minimal to-do app without distractions, just your workweek with things to do at a glance. Free for one year if you invite friends.

Check it out


Collective #561 was written by Pedro Botelho and published on Codrops.


Source: Codrops, Collective #561

Writing Asynchronous Tasks In Modern JavaScript

dreamt up by webguru in Uncategorized | Comments Off on Writing Asynchronous Tasks In Modern JavaScript

Writing Asynchronous Tasks In Modern JavaScript

Writing Asynchronous Tasks In Modern JavaScript

Jeremias Menichelli



JavaScript has two main characteristics as a programming language, both important to understand how our code will work. First is its synchronous nature, which means the code will run line after line, almost as you read it, and secondly that it is single-threaded, only one command is being executed at any time.

As the language evolved, new artifacts appeared in the scene to allow asynchronous execution; developers tried different approaches while solving more complicated algorithms and data flows, which led to the emergence of new interfaces and patterns around them.

Synchronous Execution And The Observer Pattern

As mentioned in the introduction, JavaScript runs the code you write line by line, most of the time. Even in its first years, the language had exceptions to this rule, though they were a few and you might know them already: HTTP Requests, DOM events and time intervals.

If we add an event listener to respond to the click of an element from the user, it doesn’t matter what the language interpreter is running it will stop, run the code we wrote in the listener callback and then go back to its normal flow.

Same with an interval or a network request, addEventListener, setTimeout, and XMLHttpRequest were the first artifacts to access to asynchronous execution for web developers.

Though these were exceptions of synchronous execution in JavaScript, it’s crucial to understand that the language is still single-threaded. We can break this synchronicity but the interpreter still will run one line of code at a time.

For example, let’s check out a network request.

var request = new XMLHttpRequest();
request.open('GET', '//some.api.at/server', true);

// observe for server response
request.onreadystatechange = function() {
  if (request.readyState === 4 && xhr.status === 200) {
    console.log(request.responseText);
  }
}

request.send();

No matter what is happening, by the time the server comes back, the method assigned to onreadystatechange gets called before taking back the program’s code sequence.

Something similar happens when reacting to user interaction.

const button = document.querySelector('button');

// observe for user interaction
button.addEventListener('click', function(e) {
  console.log('user click just happened!');
})

You might notice that we are hooking up to an external event and passing a callback, telling the code what to do when it takes place. Over a decade ago, “What is a callback?” was a pretty much-expected interview question because this pattern was everywhere in most codebases.

In each case mentioned, we are responding to an external event. A certain interval of time reached, a user action or a server response. We weren’t able to create an asynchronous task per se, we always observed occurrences happening outside of our reach.

This is why code shaped this way is called the Observer Pattern, which is better represented by the addEventListener interface in this case. Soon event emitters libraries or frameworks exposing this pattern flourished.

Node.js And Event Emitters

A good example is Node.js which page describes itself as “an asynchronous event-driven JavaScript runtime”, so event emitters and callback were first-class citizens. It even had an EventEmitter constructor already implemented.

const EventEmitter = require('events');
const emitter = new EventEmitter();

// respond to events
emitter.on('gretting', (message) => console.log(message));

// send events
emitter.emit('gretting', 'Hi there!');

This was not only the to-go approach for asynchronous execution but a core pattern and convention of its ecosystem. Node.js opened a new era of writing JavaScript in a different environment — even outside the web. As a consequence, other asynchronous situations were possible, like creating new directories or writing files.

const { mkdir, writeFile } = require('fs');

const styles = 'body { background: #ffdead; }';

mkdir('./assets/', (error) => {
  if (!error) {
    writeFile('assets/main.css', styles, 'utf-8', (error) => {
      if (!error) console.log('stylesheet created');
    })
  }
})

You might notice that callbacks receive an error as a first argument, if a response data is expected, it goes as a second argument. This was called Error-first Callback Pattern, which became a convention that authors and contributors adopted for their own packages and libraries.

Promises And The Endless Callback Chain

As web development faced more complex problems to solve, the need for better asynchronous artifacts appeared. If we look at the last code snippet, we can see a repeated callback chaining which doesn’t scale well as the number tasks increase.

For example, let’s add only two more steps, file reading and styles preprocessing.

const { mkdir, writeFile, readFile } = require('fs');
const less = require('less')

readFile('./main.less', 'utf-8', (error, data) => {
  if (error) throw error
  less.render(data, (lessError, output) => {
    if (lessError) throw lessError
    mkdir('./assets/', (dirError) => {
      if (dirError) throw dirError
      writeFile('assets/main.css', output.css, 'utf-8', (writeError) => {
        if (writeError) throw writeError
        console.log('stylesheet created');
      })
    })
  })
})

We can see how as the program we are writing gets more complex the code becomes harder to follow for the human eye due to multiple callback chaining and repeated error handling.

Promises, Wrappers And Chain Patterns

Promises didn’t receive much attention when they were first announced as the new addition to the JavaScript language, they aren’t a new concept as other languages had similar implementations decades before. Truth is, they turned out to change a lot the semantics and structure of most of the projects I worked on since its appearance.

Promises not only introduced a built-in solution for developers to write asynchronous code but also opened a new stage in web development serving as the construction base of later new features of the web spec like fetch.

Migrating a method from a callback approach to a promise-based one became more and more usual in projects (such as libraries and browsers), and even Node.js started slowly migrating to them.

Let’s, for example, wrap Node’s readFile method:

const { readFile } = require('fs');

const asyncReadFile = (path, options) => {
  return new Promise((resolve, reject) => {
    readFile(path, options, (error, data) => {
      if (error) reject(error);
      else resolve(data);
    })
  });
}

Here we obscure the callback by executing inside a Promise constructor, calling resolve when the method result is successful, and reject when the error object is defined.

When a method returns a Promise object we can follow its successful resolution by passing a function to then, its argument is the value which the promise was resolved, in this case, data.

If an error was thrown during the method the catch function will be called, if present.

Note: If you need to understand more in-depth how Promises work, I recommend Jake Archibald’s “JavaScript Promises: An Introduction” article which he wrote on Google’s web development blog.

Now we can use these new methods and avoid callback chains.

asyncRead('./main.less', 'utf-8')
  .then(data => console.log('file content', data))
  .catch(error => console.error('something went wrong', error))

Having a native way to create asynchronous tasks and a clear interface to follow up its possible results enabled the industry to move out of the Observer Pattern. Promise-based ones seemed to solve the unreadable and prone-to-error code.

As a better syntax highlighting or clearer error messages help while coding, a code that is easier to reason becomes more predictable for the developer reading it, with a better picture of the execution path the easier to catch a possible pitfall.

Promises adoption was so global in the community that Node.js rapidly release built-in versions of its I/O methods to return Promise objects like importing them file operations from fs.promises.

It even provided a promisify util to wrap any function which followed the Error-first Callback Pattern and transform it into a Promise-based one.

But do Promises help in all cases?

Let’s re-imagine our style preprocessing task written with Promises.

const { mkdir, writeFile, readFile } = require('fs').promises;
const less = require('less')

readFile('./main.less', 'utf-8')
  .then(less.render)
  .then(result =>
    mkdir('./assets')
      .then(writeFile('assets/main.css', result.css, 'utf-8'))
  )
  .catch(error => console.error(error))

There is a clear reduction of redundancy in the code, especially around the error handling as we now rely on catch, but Promises somehow failed to deliver a clear code indentation that directly relates to the concatenation of actions.

This is actually achieved on the first then statement after readFile is called. What happens after these lines is the need to create a new scope where we can first make the directory, to later write the result in a file. This causes a break into the indentation rhythm, not making it easy to determinate the instructions sequence at first glance.

A way to solve this is to pre-baked a custom method that handles this and allows the correct concatenation of the method, but we would be introducing one more depth of complexity to a code that already seems to have what it needs to achieve the task we want.

Note: Take in count this is an example program, and we are in control around some of the methods and they all follow an industry convention, but that’s not always the case. With more complex concatenations or the introduction of a library with a different shape, our code style can easily break.

Gladly, the JavaScript community learned again from other language syntaxes and added a notation that helps a lot around these cases where asynchronous tasks concatenation is not as pleasant or straight-forward to read as synchronous code is.

Async And Await

A Promise is defined as an unresolved value at execution time, and creating an instance of a Promise is an explicit call of this artifact.

const { mkdir, writeFile, readFile } = require('fs').promises;
const less = require('less')

readFile('./main.less', 'utf-8')
  .then(less.render)
  .then(result =>
    mkdir('./assets')
      .then(writeFile('assets/main.css', result.css, 'utf-8'))
  )
  .catch(error => console.error(error))

Inside an async method, we can use the await reserved word to determinate the resolution of a Promise before continuing its execution.

Let’s revisit or code snippet using this syntax.

const { mkdir, writeFile, readFile } = require('fs').promises;
const less = require('less')

async function processLess() {
  const content = await readFile('./main.less', 'utf-8')
  const result = await less.render(content)
  await mkdir('./assets')
  await writeFile('assets/main.css', result.css, 'utf-8')
}

processLess()

Note: Notice that we needed to move all our code to a method because we can’t use await outside the scope of an async function today.

Every time an async method finds an await statement, it will stop executing until the proceeding value or promise gets resolved.

There’s a clear consequence of using async/await notation, despite its asynchronous execution, the code looks as if it was synchronous, which is something we developers are more used to see and reason around.

What about error handling? For it, we use statements that have been present for a long time in the language, try and catch.

const { mkdir, writeFile, readFile } = require('fs').promises;
const less = require('less')

async function processLess() {
  const content = await readFile('./main.less', 'utf-8')
  const result = await less.render(content)
  await mkdir('./assets')
  await writeFile('assets/main.css', result.css, 'utf-8')
}

try {
  processLess()
} catch (e) {
  console.error(e)
}

We rest assured any error thrown in the process will be handled by the code inside the catch statement. We have a centric place that takes care of error handling, but now we have a code that is easier to read and follow.

Having consequent actions that returned value doesn’t need to be stored in variables like mkdir that don’t break the code rhythm; there’s also no need to create a new scope to access the value of result in a later step.

It’s safe to say Promises were a fundamental artifact introduced in the language, necessary to enable async/await notation in JavaScript, which you can use on both modern browsers and latest versions of Node.js.

Note: Recently in JSConf, Ryan Dahl, creator and first contributor of Node, regretted not sticking to Promises on its early development mostly because the goal of Node was to create event-driven servers and file management which the Observer pattern served better for.

Conclusion

The introduction of Promises into the web development world came to change the way we queue actions in our code and changed how we reason about our code execution and how we author libraries and packages.

But moving away from chains of callback is harder to solve, I think that having to pass a method to then didn’t help us to move away from the train of thought after years of being accustomed to the Observer Pattern and approaches adopted by major vendors in the community like Node.js.

As Nolan Lawson says in his excellent article about wrong uses in Promise concatenations, old callback habits die hard! He later explains how to escape some of these pitfalls.

I believe Promises were needed as a middle step to allow a natural way to generate asynchronous tasks but didn’t help us much to move forward on better code patterns, sometimes you actually need a more adaptable and improved language syntax.


As we try to solve more complex puzzles using JavaScript, we see the need for a more mature language and we experiment with architectures and patterns we weren’t used to seeing on the web before.

We still don’t know how the ECMAScript spec will look in years as we are always extending the JavaScript governance outside the web and try to solve more complicated puzzles.

It’s hard to say now what exactly we will need from the language for some of these puzzles to turn into simpler programs, but I’m happy with how the web and JavaScript itself are moving things, trying to adapt to challenges and new environments. I feel right now JavaScript is a more asynchronous friendly place than when I started writing code in a browser over a decade ago.

Further Reading

Smashing Editorial
(dm, il)

Source: Smashing Magazine, Writing Asynchronous Tasks In Modern JavaScript

Collective #560

dreamt up by webguru in Uncategorized | Comments Off on Collective #560


divi4-template-areas-FE

Our Sponsor

Divi 4.0 Has Arrived!

Divi 4.0 has arrived and the new fully-featured website templating system allows you to use the Divi Builder to structure your website and edit any part of the Divi Theme including headers, footers, post templates, category templates and more.

Check it out









C560_portfolio

Bruno Simon

A wonderful portfolio site of creative developer Bruno Simon built with Three.js and Cannon.js.

Check it out


C560_regex

RegexGuide

The RegexGuide is a playground helping developers to start writing regular expressions.

Check it out


C560_fuzz

Jsfuzz

Jsfuzz is coverage-guided fuzzer for testing JavaScript/Node.js packages.

Check it out




C560_platform

Using the Platform

Tim Kadlec on the incredible amount of thought and care that go into deciding the future of the web platform.

Read it






Collective #560 was written by Pedro Botelho and published on Codrops.


Source: Codrops, Collective #560

Create A Bookmarking Application With FaunaDB, Netlify And 11ty

dreamt up by webguru in Uncategorized | Comments Off on Create A Bookmarking Application With FaunaDB, Netlify And 11ty

Create A Bookmarking Application With FaunaDB, Netlify And 11ty

Create A Bookmarking Application With FaunaDB, Netlify And 11ty

Bryan Robinson



The JAMstack (JavaScript, APIs and Markup) revolution is in full swing. Static sites are secure, fast, reliable and fun to work on. At the heart of the JAMstack are static site generators (SSGs) that store your data as flat files: Markdown, YAML, JSON, HTML, and so on. Sometimes, managing data this way can be overly complicated. Sometimes, we still need a database.

With that in mind, Netlify — a static site host and FaunaDB — a serverless cloud database — collaborated to make combining both systems easier. 

Why A Bookmarking Site?

The JAMstack is great for many professional uses, but one of my favorite aspects of this set of technology is its low barrier to entry for personal tools and projects.

There are plenty of good products on the market for most applications I could come up with, but none would be exactly set up for me. None would give me full control over my content. None would come without a cost (monetary or informational).

With that in mind, we can create our own mini-services using JAMstack methods. In this case, we’ll be creating a site to store and publish any interesting articles I come across in my daily technology reading.

I spend a lot of time reading articles that have been shared on Twitter. When I like one, I hit the “heart” icon. Then, within a few days, it’s nearly impossible to find with the influx of new favorites. I want to build something as close to the ease of the “heart,” but that I own and control.

How are we going to do that? I’m glad you asked.

Interested in getting the code? You can grab it on Github or just deploy straight to Netlify from that repository! Take a look at the finished product here.

Our Technologies

Hosting And Serverless Functions: Netlify

For hosting and serverless functions, we’ll be utilizing Netlify. As an added bonus, with the new collaboration mentioned above, Netlify’s CLI — “Netlify Dev” — will automatically connect to FaunaDB and store our API keys as environment variables.

Database: FaunaDB

FaunaDB is a “serverless” NoSQL database. We’ll be using it to store our bookmarks data.

Static Site Generator: 11ty

I’m a big believer in HTML. Because of this, the tutorial won’t be using front-end JavaScript to render our bookmarks. Instead, we’ll utilize 11ty as a static site generator. 11ty has built-in data functionality that makes fetching data from an API as easy as writing a couple of short JavaScript functions.

iOS Shortcuts

We’ll need an easy way to post data to our database. In this case, we’ll use iOS’s Shortcuts app. This could be converted to an Android or desktop JavaScript bookmarklet, as well.

Setting Up FaunaDB Via Netlify Dev

Whether you have already signed up for FaunaDB or you need to create a new account, the easiest way to set up a link between FaunaDB and Netlify is via Netlify’s CLI: Netlify Dev. You can find full instructions from FaunaDB here or follow along below.

Netlify Dev running in the final project with our environment variable names showing

Netlify Dev running in the final project with our environment variable names showing (Large preview)

If you don’t already have this installed, you can run the following command in Terminal:

npm install netlify-cli -g

From within your project directory, run through the following commands:

netlify init // This will connect your project to a Netlify project

netlify addons:create fauna // This will install the FaunaDB "addon"

netlify addons:auth fauna // This command will run you through connecting your account or setting up an account

Once this is all connected, you can run netlify dev in your project. This will run any build scripts we set up, but also connect to the Netlify and FaunaDB services and grab any necessary environment variables. Handy!

Creating Our First Data

From here, we’ll log into FaunaDB and create our first data set. We’ll start by creating a new Database called “bookmarks.” Inside a Database, we have Collections, Documents and Indexes.

A screenshot of the FaunaDB console with data

A screenshot of the FaunaDB console with data (Large preview)

A Collection is a categorized group of data. Each piece of data takes the form of a Document. A Document is a “single, changeable record within a FaunaDB database,” according to Fauna’s documentation. You can think of Collections as a traditional database table and a Document as a row.

For our application, we need one Collection, which we’ll call “links.” Each document within the “links” Collection will be a simple JSON object with three properties. To start, we’ll add a new Document that we’ll use to build our first data fetch.

{
  "url": "https://css-irl.info/debugging-css-grid-part-2-what-the-fraction/",
  "pageTitle": "CSS { In Real Life } | Debugging CSS Grid – Part 2: What the Fr(action)?",
  "description": "CSS In Real Life is a blog covering CSS topics and useful snippets on the web’s most beautiful language. Published by Michelle Barker, front end developer at Ordoo and CSS superfan."
}

This creates the basis for the information we’ll need to pull from our bookmarks as well as provides us with our first set of data to pull into our template.

If you’re like me, you want to see the fruits of your labor right away. Let’s get something on the page!

Installing 11ty And Pulling Data Into A Template

Since we want the bookmarks to be rendered in HTML and not fetched by the browser, we’ll need something to do the rendering. There are many great ways of doing it, but for ease and power, I love using the 11ty static site generator.

Since 11ty is a JavaScript static site generator, we can install it via NPM.

npm install --save @11ty/eleventy

From that installation, we can run eleventy or eleventy --serve in our project to get up and running.

Netlify Dev will often detect 11ty as a requirement and run the command for us. To have this work – and make sure we’re ready to deploy, we can also create “serve” and “build” commands in our package.json.

"scripts": {
    "build": "npx eleventy",
    "serve": "npx eleventy --serve"
  }

11ty’s Data Files

Most static site generators have an idea of a “data file” built-in. Usually, these files will be JSON or YAML files that allow you to add extra information to your site.

In 11ty, you can use JSON data files or JavaScript data files. By utilizing a JavaScript file, we can actually make our API calls and return the data directly into a template.

The file will be a JavaScript module. So in order to have anything work, we need to export either our data or a function. In our case, we’ll export a function.

module.exports = async function() {  
    const data = mapBookmarks(await getBookmarks());  

    return data.reverse()  
}

Let’s break that down. We have two functions doing our main work here: mapBookmarks() and getBookmarks()

The getBookmarks() function will go fetch our data from our FaunaDB database and mapBookmarks() will take an array of bookmarks and restructure it to work better for our template.

Let’s dig deeper into getBookmarks().

getBookmarks()

First, we’ll need to install and initialize an instance of the FaunaDB JavaScript driver.

npm install --save faunadb

Now that we’ve installed it, let’s add it to the top of our data file. This code is straight from Fauna’s docs.

// Requires the Fauna module and sets up the query module, which we can use to create custom queries.
const faunadb = require('faunadb'),  
      q = faunadb.query;

// Once required, we need a new instance with our secret
var adminClient = new faunadb.Client({  
   secret: process.env.FAUNADB_SERVER_SECRET  
});

After that, we can create our function. We’ll start by building our first query using built-in methods on the driver. This first bit of code will return the database references we can use to get full data for all of our bookmarked links. We use the Paginate method, as a helper to manage cursor state should we decide to paginate the data before handing it to 11ty. In our case, we’ll just return all the references.

In this example, I’m assuming you installed and connected FaunaDB via the Netlify Dev CLI. Using this process, you get local environment variables of the FaunaDB secrets. If you didn’t install it this way or aren’t running netlify dev in your project, you’ll need a package like dotenv to create the environment variables. You’ll also need to add your environment variables to your Netlify site configuration to make deploys work later.

adminClient.query(q.Paginate(  
       q.Match( // Match the reference below
           q.Ref("indexes/all_links") // Reference to match, in this case, our all_links index  
       )  
   ))  
   .then( response => { ... })

This code will return an array of all of our links in reference form. We can now build a query list to send to our database.

adminClient.query(...)
    .then((response) => {  
        const linkRefs = response.data; // Get just the references for the links from the response 
        const getAllLinksDataQuery = linkRefs.map((ref) => {  
        return q.Get(ref) // Return a Get query based on the reference passed in  
   })  

return adminClient.query(getAllLinksDataQuery).then(ret => {  
    return ret // Return an array of all the links with full data  
       })  
   }).catch(...)

From here, we just need to clean up the data returned. That’s where mapBookmarks() comes in!

mapBookmarks()

In this function, we deal with two aspects of the data.

First, we get a free dateTime in FaunaDB. For any data created, there’s a timestamp (ts) property. It’s not formatted in a way that makes Liquid’s default date filter happy, so let’s fix that.

function mapBookmarks(data) {
    return data.map(bookmark => {
        const dateTime = new Date(bookmark.ts / 1000);
        ...
    })
}

With that out of the way, we can build a new object for our data. In this case, it will have a time property, and we’ll use the Spread operator to destructure our data object to make them all live at one level.

function mapBookmarks(data) {
    return data.map(bookmark => {
        const dateTime = new Date(bookmark.ts / 1000);

        return { time: dateTime, ...bookmark.data }
    })
}

Here’s our data before our function:

{ 
  ref: Ref(Collection("links"), "244778237839802888"),
  ts: 1569697568650000,
  
  data: { 
    url: 'https://sample.com',
    pageTitle: 'Sample title',
    description: 'An escaped description goes here' 
  } 
}

Here’s our data after our function:

{
    time: 1569697568650,
    url: 'https://sample.com',
    pageTitle: 'Sample title'
    description: 'An escaped description goes here'
}

Now, we’ve got well-formatted data that’s ready for our template!

Let’s write a simple template. We’ll loop through our bookmarks and validate that each has a pageTitle and a url so we don’t look silly.

{% for link in bookmarks %} {% if link.url and link.pageTitle %} // confirms there’s both title AND url for safety

{{ link.pageTitle }}

Saved on {{ link.time | date: "%b %d, %Y" }}

{% if link.description != "" %}

{{ link.description }}

{% endif %}
{% endif %} {% endfor %} </div>

We’re now ingesting and displaying data from FaunaDB. Let’s take a moment and think about how nice it is that this renders out pure HTML and there’s no need to fetch data on the client side!

But that’s not really enough to make this a useful app for us. Let’s figure out a better way than adding a bookmark in the FaunaDB console.

Enter Netlify Functions

Netlify’s Functions add-on is one of the easier ways to deploy AWS lambda functions. Since there’s no configuration step, it’s perfect for DIY projects where you just want to write the code.

This function will live at a URL in your project that looks like this: https://myproject.com/.netlify/functions/bookmarks assuming the file we create in our functions folder is bookmarks.js.

Basic Flow

  1. Pass a URL as a query parameter to our function URL.
  2. Use the function to load the URL and scrape the page’s title and description if available.
  3. Format the details for FaunaDB.
  4. Push the details to our FaunaDB Collection.
  5. Rebuild the site.

Requirements

We’ve got a few packages we’ll need as we build this out. We’ll use the netlify-lambda CLI to build our functions locally. request-promise is the package we’ll use for making requests. Cheerio.js is the package we’ll use to scrape specific items from our requested page (think jQuery for Node). And finally, we’ll need FaunaDb (which should already be installed.

npm install --save netlify-lambda request-promise cheerio

Once that’s installed, let’s configure our project to build and serve the functions locally.

We’ll modify our “build” and “serve” scripts in our package.json to look like this:

"scripts": {
    "build": "npx netlify-lambda build lambda --config ./webpack.functions.js && npx eleventy",
    "serve": "npx netlify-lambda build lambda --config ./webpack.functions.js && npx eleventy --serve"
}

Warning: There’s an error with Fauna’s NodeJS driver when compiling with Webpack, which Netlify’s Functions use to build. To get around this, we need to define a configuration file for Webpack. You can save the following code to a newor existingwebpack.config.js.

const webpack = require('webpack');

module.exports = {
  plugins: [ new webpack.DefinePlugin({ "global.GENTLY": false }) ]
};

Once this file exists, when we use the netlify-lambda command, we’ll need to tell it to run from this configuration. This is why our “serve” and “build scripts use the --config value for that command.

Function Housekeeping

In order to keep our main Function file as clean as possible, we’ll create our functions in a separate bookmarks directory and import them into our main Function file.

import { getDetails, saveBookmark } from "./bookmarks/create";

getDetails(url)

The getDetails() function will take a URL, passed in from our exported handler. From there, we’ll reach out to the site at that URL and grab relevant parts of the page to store as data for our bookmark.

We start by requiring the NPM packages we need:

const rp = require('request-promise');  
const cheerio = require('cheerio');

Then, we’ll use the request-promise module to return an HTML string for the requested page and pass that into cheerio to give us a very jQuery-esque interface.

const getDetails = async function(url) {  
    const data = rp(url).then(function(htmlString) {  
        const $ = cheerio.load(htmlString);  
        ...  
}

From here, we need to get the page title and a meta description. To do that, we’ll use selectors like you would in jQuery. 

Note: In this code, we use 'head > title' as the selector to get the title of the page. If you don’t specify this, you may end up getting <title> tags inside of all SVGs on the page, which is less than ideal.

const getDetails = async function(url) {
  const data = rp(url).then(function(htmlString) {
    const $ = cheerio.load(htmlString);
    const title = $('head > title').text(); // Get the text inside the tag  
    const description = $('meta[name="description"]').attr('content'); // Get the text of the content attribute

// Return out the data in the structure we expect  
    return {
      pageTitle: title,
      description: description
    };
  });
  return data //return to our main function  
}

With data in hand, it’s time to send our bookmark off to our Collection in FaunaDB!

saveBookmark(details)

For our save function, we’ll want to pass the details we acquired from getDetails as well as the URL as a singular object. The Spread operator strikes again!

const savedResponse = await saveBookmark({url, ...details});

In our create.js file, we also need to require and setup our FaunaDB driver. This should look very familiar from our 11ty data file.

const faunadb = require('faunadb'),  
      q = faunadb.query;  

const adminClient = new faunadb.Client({  
   secret: process.env.FAUNADB_SERVER_SECRET  
});

Once we’ve got that out of the way, we can code.

First, we need to format our details into a data structure that Fauna is expecting for our query. Fauna expects an object with a data property containing the data we wish to store.

const saveBookmark = async function(details) {  
const data = {  
   data: details  
};

...

}

Then we’ll open a new query to add to our Collection. In this case, we’ll use our query helper and use the Create method. Create() takes two arguments. First is the Collection in which we want to store our data and the second is the data itself.

After we save, we return either success or failure to our handler.

const saveBookmark = async function(details) {  
const data = {  
   data: details  
};

return adminClient.query(q.Create(q.Collection("links"), data))  
   .then((response) => {  
        /* Success! return the response with statusCode 200 */  
        return {  
             statusCode: 200,  
             body: JSON.stringify(response)  
         }  
     }).catch((error) => {  
        /* Error! return the error with statusCode 400 */  
        return  {  
             statusCode: 400,  
             body: JSON.stringify(error)  
         }  
     })  
}

Let’s take a look at the full Function file.

import { getDetails, saveBookmark } from "./bookmarks/create";  
import { rebuildSite } from "./utilities/rebuild"; // For rebuilding the site (more on that in a minute)

exports.handler = async function(event, context) {  
    try {  
        const url = event.queryStringParameters.url; // Grab the URL  

        const details = await getDetails(url); // Get the details of the page  
        const savedResponse = await saveBookmark({url, ...details}); //Save the URL and the details to Fauna  

        if (savedResponse.statusCode === 200) { 
            // If successful, return success and trigger a Netlify build  
            await rebuildSite();  
            return { statusCode: 200, body: savedResponse.body }  
         } else {  
            return savedResponse //or else return the error  
         }  
     } catch (err) {  
        return { statusCode: 500, body: `Error: ${err}` };  
     }  
};

rebuildSite()

The discerning eye will notice that we have one more function imported into our handler: rebuildSite(). This function will use Netlify’s Deploy Hook functionality to rebuild our site from the new data every time we submit a new — successful — bookmark save.

In your site’s settings in Netlify, you can access your Build & Deploy settings and create a new “Build Hook.” Hooks have a name that appears in the Deploy section and an option for a non-master branch to deploy if you so wish. In our case, we’ll name it “new_link” and deploy our master branch.

A visual reference for the Netlify Admin’s build hook setup

A visual reference for the Netlify Admin’s build hook setup (Large preview)

From there, we just need to send a POST request to the URL provided.

We need a way of making requests and since we’ve already installed request-promise, we’ll continue to use that package by requiring it at the top of our file.

const rp = require('request-promise');  

const rebuildSite = async function() {  
    var options = {  
         method: 'POST',  
         uri: 'https://api.netlify.com/build_hooks/5d7fa6175504dfd43377688c',  
         body: {},  
         json: true  
    };  

    const returned = await rp(options).then(function(res) {  
         console.log('Successfully hit webhook', res);  
     }).catch(function(err) {  
         console.log('Error:', err);  
     });  

    return returned  
}
A demo of the Netlify Function setup and the iOS Shortcut setup combined

Setting Up An iOS Shortcut

So, we have a database, a way to display data and a function to add data, but we’re still not very user-friendly.

Netlify provides URLs for our Lambda functions, but they’re not fun to type into a mobile device. We’d also have to pass a URL as a query parameter into it. That’s a LOT of effort. How can we make this as little effort as possible?

A visual reference for the setup for our Shortcut functionality

A visual reference for the setup for our Shortcut functionality (Large preview)

Apple’s Shortcuts app allows the building of custom items to go into your share sheet. Inside these shortcuts, we can send various types of requests of data collected in the share process.

Here’s the step-by-step Shortcut:

  1. Accept any items and store that item in a “text” block.
  2. Pass that text into a “Scripting” block to URL encode (just in case).
  3. Pass that string into a URL block with our Netlify Function’s URL and a query parameter of url.
  4. From “Network” use a “Get contents” block to POST to JSON to our URL.
  5. Optional: From “Scripting” “Show” the contents of the last step (to confirm the data we’re sending).

To access this from the sharing menu, we open up the settings for this Shortcut and toggle on the “Show in Share Sheet” option.

As of iOS13, these share “Actions” are able to be favorited and moved to a high position in the dialog.

We now have a working “app” for sharing bookmarks across multiple platforms!

Go The Extra Mile!

If you’re inspired to try this yourself, there are a lot of other possibilities to add functionality. The joy of the DIY web is that you can make these sorts of applications work for you. Here are a few ideas:

  1. Use a faux “API key” for quick authentication, so other users don’t post to your site (mine uses an API key, so don’t try to post to it!).
  2. Add tag functionality to organize bookmarks.
  3. Add an RSS feed for your site so that others can subscribe.
  4. Send out a weekly roundup email programmatically for links that you’ve added.

Really, the sky is the limit, so start experimenting!

Smashing Editorial
(dm, yk)

Source: Smashing Magazine, Create A Bookmarking Application With FaunaDB, Netlify And 11ty

Writing A Multiplayer Text Adventure Engine In Node.js: Game Engine Server Design (Part 2)

dreamt up by webguru in Uncategorized | Comments Off on Writing A Multiplayer Text Adventure Engine In Node.js: Game Engine Server Design (Part 2)

Writing A Multiplayer Text Adventure Engine In Node.js: Game Engine Server Design (Part 2)

Writing A Multiplayer Text Adventure Engine In Node.js: Game Engine Server Design (Part 2)

Fernando Doglio



After some careful consideration and actual implementation of the module, some of the definitions I made during the design phase had to be changed. This should be a familiar scene for anyone who has ever worked with an eager client who dreams about an ideal product but needs to be restraint by the development team.

Once features have been implemented and tested, your team will start noticing that some characteristics might differ from the original plan, and that’s alright. Simply notify, adjust, and go on. So, without further ado, allow me to first explain what has changed from the original plan.

Battle Mechanics

This is probably the biggest change from the original plan. I know I said I was going to go with a D&D-esque implementation in which each PC and NPC involved would get an initiative value and after that, we would run a turn-based combat. It was a nice idea, but implementing it on a REST-based service is a bit complicated since you can’t initiate the communication from the server side, nor maintain status between calls.

So instead, I will take advantage of the simplified mechanics of REST and use that to simplify our battle mechanics. The implemented version will be player-based instead of party-based, and will allow players to attack NPCs (Non-Player Characters). If their attack succeeds, the NPCs will be killed or else they will attack back by either damaging or killing the player.

Whether an attack succeeds or fails will be determined by the type of weapon used and the weaknesses an NPC might have. So basically, if the monster you’re trying to kill is weak against your weapon, it dies. Otherwise, it’ll be unaffected and — most likely — very angry.

Triggers

If you paid close attention to the JSON game definition from my previous article, you might’ve noticed the trigger’s definition found on scene items. A particular one involved updating the game status (statusUpdate). During implementation, I realized having it working as a toggle provided limited freedom. You see, in the way it was implemented (from an idiomatic point of view), you were able to set a status but unsetting it wasn’t an option. So instead, I’ve replaced this trigger effect with two new ones: addStatus and removeStatus. These will allow you to define exactly when these effects can take place — if at all. I feel this is a lot easier to understand and reason about.

This means that the triggers now look like this:

"triggers": [
{
    "action": "pickup",
"effect":{
    "addStatus": "has light",
"target": "game"
    }
},
{
    "action": "drop",
    "effect": {
    "removeStatus": "has light",
    "target": "game"
    }
}
]

When picking up the item, we’re setting up a status, and when dropping it, we’re removing it. This way, having multiple game-level status indicators is completely possible and easy to manage.

The Implementation

With those updates out of the way, we can start covering the actual implementation. From an architectural point of view, nothing changed; we’re still building a REST API that will contain the main game engine’s logic.

The Tech Stack

For this particular project, the modules I’m going to be using are the following:

Module Description
Express.js Obviously, I’ll be using Express to be the base for the entire engine.
Winston Everything in regards to logging will be handled by Winston.
Config Every constant and environment-dependant variable will be handled by the config.js module, which greatly simplifies the task of accessing them.
Mongoose This will be our ORM. I will model all resources using Mongoose Models and use that to interact directly with the database.
uuid We’ll need to generate some unique IDs — this module will help us with that task.

As for other technologies used aside from Node.js, we have MongoDB and Redis. I like to use Mongo due to the lack of schema required. That simple fact allows me to think about my code and the data formats, without having to worry about updating the structure of my tables, schema migrations or conflicting data types.

Regarding Redis, I tend to use it as a support system as much as I can in my projects and this case is no different. I will be using Redis for everything that can be considered volatile information, such as party member numbers, command requests, and other types of data that are small enough and volatile enough to not merit permanent storage.

I’m also going to be using Redis’ key expiration feature to auto manage some aspects of the flow (more on this shortly).

API Definition

Before moving into client-server interaction and data-flow definitions I want to go over the endpoints defined for this API. They aren’t that many, mostly we need to comply with the main features described in Part 1:

Feature Description
Join a game A player will be able to join a game by specifying the game’s ID.
Create a new game A player can also create a new game instance. The engine should return an ID, so that others can use it to join.
Return scene This feature should return the current scene where the party is located. Basically, it’ll return the description, with all of the associated information (possible actions, objects in it, etc.).
Interact with scene This is going to be one of the most complex ones, because it will take a command from the client and perform that action — things like move, push, take, look, read, to name just a few.
Check inventory Although this is a way to interact with the game, it does not directly relate to the scene. So, checking the inventory for each player will be considered a different action.
Register client application The above actions require a valid client to execute them. This endpoint will verify the client application and return a Client ID that will be used for authentication purposes on subsequent requests.

The above list translates into the following list of endpoints:

Verb Endpoint Description
POST /clients Client applications will require to get a Client ID key using this endpoint.
POST /games New game instances are created using this endpoint by the client applications.
POST /games/:id Once the game is created, this endpoint will enable party members to join it and start playing.
GET /games/:id/:playername This endpoint will return the current game state for a particular player.
POST /games/:id/:playername/commands Finally, with this endpoint, the client application will be able to submit commands (in other words, this endpoint will be used to play).

Let me go into a bit more detail about some of the concepts I described in the previous list.

Client Apps

The client applications will need to register into the system to start using it. All endpoints (except for the first one on the list) are secured and will require a valid application key to be sent with the request. In order to obtain that key, client apps need to simply request one. Once provided, they will last for as long as they are used, or will expire after a month of not being used. This behavior is controlled by storing the key in Redis and setting a one-month long TTL to it.

Game Instance

Creating a new game basically means creating a new instance of a particular game. This new instance will contain a copy of all of the scenes and their content. Any modifications done to the game will only affect the party. This way, many groups can play the same game on their own individual way.

Player’s Game State

This is similar to the previous one, but unique to each player. While the game instance holds the game state for the entire party, the player’s game state holds the current status for one particular player. Mainly, this holds inventory, position, current scene and HP (health points).

Player Commands

Once everything is set up and the client application has registered and joined a game, it can start sending commands. The implemented commands in this version of the engine include: move, look, pickup and attack.

  • The move command will allow you to traverse the map. You’ll be able to specify the direction you want to move towards and the engine will let you know the result. If you take a quick glimpse at Part 1, you can see the approach I took to handle maps. (In short, the map is represented as a graph, where each node represents a room or scene and is only connected to other nodes that represent adjacent rooms.)

    The distance between nodes is also present in the representation and coupled with the standard speed a player has; going from room to room might not be as simple as stating your command, but you’ll also have to traverse the distance. In practice, this means that going from one room to the other might require several move commands). The other interesting aspect of this command comes from the fact that this engine is meant to support multiplayer parties, and the party can’t be split (at least not at this time).

    Therefore, the solution for this is similar to a voting system: every party member will send a move command request whenever they want. Once more than half of them have done so, the most requested direction will be used.

  • look is quite different from move. It allows the player to specify a direction, an item or NPC they want to inspect. The key logic behind this command, comes into consideration when you think about status-dependant descriptions.

    For example, let’s say that you enter a new room, but it’s completely dark (you don’t see anything), and you move forward while ignoring it. A few rooms later, you pick up a lit torch from a wall. So now you can go back and re-inspect that dark room. Since you’ve picked up the torch, you now can see inside of it, and be able to interact with any of the items and NPCs you find in there.

    This is achieved by maintaining a game wide and player specific set of status attributes and allowing the game creator to specify several descriptions for our status-dependant elements in the JSON file. Every description is then equipped with a default text and a set of conditional ones, depending on the current status. The latter are optional; the only one that is mandatory is the default value.

    Additionally, this command has a short-hand version for look at room: look around; that is because players will be trying to inspect a room very often, so providing a short-hand (or alias) command that is easier to type makes a lot of sense.

  • The pickup command plays a very important role for the gameplay. This command takes care of adding items into the players inventory or their hands (if they’re free). In order to understand where each item is meant to be stored, their definition has a “destination” property that specifies if it is meant for the inventory or the player’s hands. Anything that is successfully picked up from the scene is then removed from it, updating the game instance’s version of the game.
  • The use command will allow you to affect the environment using items in your inventory. For instance, picking up a key in a room will allow you to use it to open a locked door in another room.
  • There is a special command, one that is not gameplay-related, but instead a helper command meant to obtain particular information, such as the current game ID or the player’s name. This command is called get, and the players can use it to query the game engine. For example: get gameid.
  • Finally, the last command implemented for this version of the engine is the attack command. I already covered this one; basically, you’ll have to specify your target and the weapon you’re attacking it with. That way the system will be able to check the target’s weaknesses and determine the output of your attack.

Client-Engine Interaction

In order to understand how to use the above-listed endpoints, let me show you how any would-be-client can interact with our new API.

Step Description
Register client First things first, the client application needs to request an API key to be able to access all other endpoints. In order to get that key, it needs to register on our platform. The only parameter to provide is the name of the app, that’s all.
Create a game After the API key is obtained, the first thing to do (assuming this is a brand new interaction) is to create a brand new game instance. Think about it this way: the JSON file I created in my last post contains the game’s definition, but we need to create an instance of it just for you and your party (think classes and objects, same deal). You can do with that instance whatever you want, and it will not affect other parties.
Join the game After creating the game, you’ll get a game ID back from the engine. You can then use that game ID to join the instance using your unique username. Unless you join the game, you can’t play, because joining the game will also create a game state instance for you alone. This will be where your inventory, your position and your basic stats are saved in relation to the game you’re playing. You could potentially be playing several games at the same time, and in each one have independent states.
Send commands In other words: play the game. The final step is to start sending commands. The amount of commands available was already covered, and it can be easily extended (more on this in a bit). Everytime you send a command, the game will return the new game state for your client to update your view accordingly.

Let’s Get Our Hands Dirty

I’ve gone over as much design as I can, in the hopes that that information will help you understand the following part, so let’s get into the nuts and bolts of the game engine.

Note: I will not be showing you the full code in this article since it’s quite big and not all of it is interesting. Instead, I’ll show the more relevant parts and link to the full repository in case you want more details.

The Main File

First things first: this is an Express project and it’s based boilerplate code was generated using Express’ own generator, so the app.js file should be familiar to you. I just want to go over two tweaks I like to do on that code to simplify my work.

First, I add the following snippet to automate the inclusion of new route files:

const requireDir = require("require-dir")
const routes = requireDir("./routes")

//...

Object.keys(routes).forEach( (file) => {
    let cnt = routes[file]
    app.use('/' + file, cnt)    
})

It’s quite simple really, but it removes the need to manually require each route files you create in the future. By the way, require-dir is a simple module that takes care of auto-requiring every file inside a folder. That’s it.

The other change I like to do is to tweak my error handler just a little bit. I should really start using something more robust, but for the needs at hand, I feel like this gets the work done:

// error handler
app.use(function(err, req, res, next) {
  // render the error page
  if(typeof err === "string") {
    err = {
      status: 500,
      message: err
    }
  }
  res.status(err.status || 500);
  let errorObj = {
    error: true,
    msg: err.message,
    errCode: err.status || 500
  }
  if(err.trace) {
    errorObj.trace = err.trace
  }

  res.json(errorObj);
});

The above code takes care of the different types of error messages we might have to deal with — either full objects, actual error objects thrown by Javascript or simple error messages without any other context. This code will take it all and format it into a standard format.

Handling Commands

This is another one of those aspects of the engine that had to be easy to extend. In a project like this one, it makes total sense to assume new commands will pop up in the future. If there is something you want to avoid, then that would probably be avoid making changes on the base code when trying to add something new three or four months in the future.

No amount of code comments will make the task of modifying code you haven’t touched (or even thought about) in several months easy, so the priority is to avoid as many changes as possible. Lucky for us, there are a few patterns we can implement to solve this. In particular, I used a mixture of the Command and the Factory patterns.

I basically encapsulated the behavior of each command inside a single class which inherits from a BaseCommand class that contains the generic code to all commands. At the same time, I added a CommandParser module that grabs the string sent by the client and returns the actual command to execute.

The parser is very simple since all implemented commands now have the actual command as to their first word (i.e. “move north”, “pick up knife”, and so on) it’s a simple matter of splitting the string and getting the first part:

const requireDir = require("require-dir")
const validCommands = requireDir('./commands')

class CommandParser {


    constructor(command) {
        this.command = command
    }


    normalizeAction(strAct) {
        strAct = strAct.toLowerCase().split(" ")[0]
        return strAct
    }


    verifyCommand() {
        if(!this.command) return false
        if(!this.command.action) return false
        if(!this.command.context) return false

        let action = this.normalizeAction(this.command.action)

        if(validCommands[action]) {
            return validCommands[action]
        }
        return false
    }

    parse() {
        let validCommand = this.verifyCommand()
        if(validCommand) {
            let cmdObj = new validCommand(this.command)
            return cmdObj
        } else {
            return false
        }
    }
}

Note: I’m using the require-dir module once again to simplify the inclusion of any existing and new command classes. I simply add it to the folder and the entire system is able to pick it up and use it.

With that being said, there are many ways this can be improved; for instance, by being able to add synonym support for our commands would be a great feature (so saying “move north”, “go north” or even “walk north” would mean the same). That is something that we could centralize in this class and affect all commands at the same time.

I won’t go into details on any of the commands because, again, that’s too much code to show here, but you can see in the following route code how I managed to generalize that handling of the existing (and any future) commands:

/**
Interaction with a particular scene
*/
router.post('/:id/:playername/:scene', function(req, res, next) {

    let command = req.body
    command.context = {
        gameId: req.params.id,
        playername: req.params.playername,
    }

    let parser = new CommandParser(command)

    let commandObj = parser.parse() //return the command instance
    if(!commandObj) return next({ //error handling
        status: 400,
          errorCode: config.get("errorCodes.invalidCommand"),
        message: "Unknown command"
    })

    commandObj.run((err, result) => { //execute the command
        if(err) return next(err)

        res.json(result)
    })

})

All commands only require the run method — anything else is extra and meant for internal use.

I encourage you to go and review the entire source code (even download it and play with it if you like!). In the next part of this series, I’ll show you the actual client implemention and interaction of this API.

Closing Thoughts

I may not have covered a lot of my code here, but I still hope that the article was helpful to show you how I tackle projects — even after the initial design phase. I feel like a lot of people try to start coding as their first response to a new idea and that sometimes can end up discouraging to a developer since there is no real plan set nor any goals to achieve — other than having the final product ready (and that is too big of a milestone to tackle from day 1). So again, my hope with these articles is to share a different way to go about working solo (or as part of a small group) on big projects.

I hope you’ve enjoyed the read! Please feel free to leave a comment below with any type of suggestions or recommendations, I’d love to read what you think and if you’re eager to start testing the API with your own client-side code.

See you on the next one!

Smashing Editorial
(dm, yk, il)

Source: Smashing Magazine, Writing A Multiplayer Text Adventure Engine In Node.js: Game Engine Server Design (Part 2)

A Guide To Optimizing Images For Mobile

dreamt up by webguru in Uncategorized | Comments Off on A Guide To Optimizing Images For Mobile

A Guide To Optimizing Images For Mobile

A Guide To Optimizing Images For Mobile

Suzanne Scacca



(This is a sponsored article.) You know how critical it is to build websites that load quickly. All it takes is for a page to load one second too long for it to start losing visitors and sales. Plus, now that Google has made mobile-first indexing the default, you really can’t afford to let any performance optimizations fall by the wayside what with how difficult it can be to get your mobile site as speedy as your desktop.

Google takes many factors into account when ranking a website and visitors take maybe a handful of factors into account when deciding to explore a site. At the intersection of the two is website speed.

It should come as no surprise that images cause a lot of the problems websites have with speed. And while you could always just trim the fat and build more minimally designed and content-centric sites, why compromise?

Images are a powerful force on the web.

Not only can well-chosen images improve the aesthetics of a site, but they also make it easier for your visitors to consume content. Of course, there are the SEO benefits of images, too.

So, today, let’s focus on how you can still design with as many images as you want without slowing down your website. This will require you to update your image optimization strategy and adopt a tool called ImageKit, but it shouldn’t take much work from you to get this new system in place.

The Necessity Of An Image Optimization Strategy For Mobile

According to HTTP Archive:

  • The median size of a desktop website in 2019 is 1939.5 KB.
  • The median size of a mobile website in 2019 is 1745.0 KB.

HTTP Archive desktop and mobile kilobytes

HTTP Archive charts how many kilobytes desktop and mobile websites are, on average. (Source: HTTP Archive) (Large preview)

If we don’t get a handle on this growth, it’s going to be impossible to meet consumer and Google demands when it comes to providing fast websites. That or we’re going to have to get really good at optimizing for speed.

Speaking of speed, let’s see what HTTP Archive has to say about image weight.

HTTP Archive image bytes desktop vs mobile

HTTP Archive plots out how much images weigh on desktop vs mobile websites. (Source: HTTP Archive) (Large preview)

As it stands today:

  • The median size of images on desktop is 980.3 KB out of the total 1939.5 KB.
  • The median size of images on mobile is 891.7 KB out of the total 1745.0 KB.

Bottom line: images add a lot of weight to websites and consume a lot of bandwidth. And although this data shows that the median size of images on mobile is less than their desktop counterparts, the proportion of images-to-website is slightly larger.

That said, if you have the right image optimization strategy in place, this can easily be remedied.

Here is what this strategy should entail:

1. Size Your Images Correctly

There are lots of tedious tasks you’d have to handle without the right automations in place. Like resizing your images.

But you have to do it, right?

Let’s say you use Unsplash to source a number of images for a website you’re working on.

Unsplash photo from Mark Boxx

An example of a photo you’d find on Unsplash, this one comes from Mark Boss. (Source: Unsplash) (Large preview)

Unlike premium stock repositories where you might get to choose what size or file format you download the file in, you don’t get a choice here.

So, you download the image and any others you need. You then have the choice to use the image as is or manually resize it. After looking at the size of the file and the dimensions of the image, you realize it would be a good idea to resize it.

Original dimensions of image from Unsplash

These are the original dimensions of the Unsplash image: 5591×3145 px. (Source: Unsplash) (Large preview)

This particular image exported as a 3.6 MB file and a 5591×3145 px image. That’s way too big for any website.

There’s no reason to upload images larger than 1 MB — and that’s even pushing it. As for dimensions? Well, that depends on the width of your site, but I think somewhere between 1200 and 2000 px should be your max.

You’re going to have to go through this same process whether images come from a stock site or from someone’s DSLR. The point is, no source image is ever going to come out the “right” size for your website, which means resizing has to take place at some point.

What’s more, responsive websites display images in different sizes depending on the device or browser they’re viewed on. And then there are the different use cases — like full-sized image vs. thumbnail or full-sized product photo vs. featured image.

So, there’s more resizing that has to be done even after you’ve gone through the trouble of manually resizing them.

Here’s what you shouldn’t do:

  • Resize images one-by-one on your own. It’s time-consuming and inefficient.
  • Rely on browser resizing to display your images responsively as it can cause issues.

Instead, you can integrate your existing image server (on your web host) or external storage service (like S3) with ImageKit. Or you can use ImageKit’s Media Library to store your files.

ImageKit Media Library upload

This is how easy it is to upload a new file to the ImageKit Media Library. (Source: ImageKit) (Large preview)

As you can see, ImageKit has accepted the upload of this Unsplash photo at its original dimensions and sizes. The same goes for wherever your files originate from.

However, once you integrate your images or image storage with ImageKit, the tool will take control of your image sizing. You can see how that’s done here:

ImageKit image URL endpoints

ImageKit image URL endpoints enable users to more easily control image resizing parameters. (Source: ImageKit) (Large preview)

Let me briefly explain what you’re looking at above:

  • The Image Origin Preference tells ImageKit where images need to be optimized from. In this case, it’s the ImageKit Media Library and they’ll be served over my website.
  • The Old Image URL is a reminder of where our images lived on the server.
  • The New Image URLs explains where your images will be optimized through ImageKit.

The formula is simple enough. You take the original URL for your image and you transform it with the new ImageKit URL.

The ImageKit URL alone will instantly shrink the size of your image files. However, if you want to do some resizing of your image’s dimensions while you’re at it, you can use transformation parameters to do so.

For example, this is the Unsplash photo as seen from the media library of my website. It lives on my own servers, which is why the address shows my own URL:

An Unsplash image without resizing

How a full-sized image from Unsplash might appear if you leave it as is on your server. (Source: Unsplash) (Large preview)

To see what it looks like once ImageKit has transformed it, I swap out my domain name with the endpoint provided by ImageKit. I then add my image resizing parameters (they allow you to do more than just resize, too) and reattach the remainder of the URL that points to my image storage.

This is what happens when I use ImageKit to automatically resize my image to 1000×560 pixels:

ImageKit endpoint image resizing

ImageKit endpoints enable users to define how their images are to be resized like in this example. (Source: ImageKit) (Large preview)

To create this resized image, I transformed the ImageKit URL into the following:

https://imagekit.io/vq1l4ywcv/tr:w-1000,h-560/…

It’s the width (w-) and height (h-) parameters that reduced the file’s dimensions.

Now, as you can see, this isn’t as pixel-perfect as the original image, but that’s because I have quite a bit of compression applied to the file (80%). I’ll cover how that works below.

In the meantime, let’s focus on how great the image still looks as well as the gains we’re about to get in speed.

ImageKit resizing example on original Unsplash photo

This is what can happen after ImageKit users resize their images. (Source: Unsplash) (Large preview)

Previously, this was a 3.6 MB file for the 5591×3145 px image. Now, it’s a 128 KB file for the 1000×560 px image.

To sweeten the deal further, ImageKit makes it easy to resize your images this way using URL-based image transformation. Essentially, it works like this:

  • You save one master image to ImageKit’s media library or your preferred server.
  • ImageKit automatically uses multiple techniques to bring down the image size significantly.
  • You can then use ImageKit’s resizing and cropping parameters to modify each image to cater to different device resolutions and sizes.

When 91mobiles took advantage of this form of image optimization, it saved its website 3.5 TB every month of bandwidth. And they didn’t have to do anything but integrate with the platform. There was no need to move their images to ImageKit or another third-party storage service. It all took place within their legacy infrastructure.

2. Use Faster-loading Image Formats

It’s not just the size of your images that drain storage space and bandwidth. The file types you use have an impact, too.

PNGs, in general, are used for things like logos, images containing text and other super-fine images that have a transparent background. While you can use them to save your photos, they tend to produce the largest sizes. Even when lossless compression is applied, PNGs still remain larger in size than other file types.

GIFs are the animated counterpart of PNGs and use lossless compression as well.

JPGs, on the other hand, are best suited for colorful images and photos. They’re smaller in size and they shrink down with lossy compression. It’s possible to compress JPGs enough to get them to a manageable size, but you have to be careful as lossy compression degrades the overall quality of a file and there’s no turning back once it’s been done.

WebPs have been gaining in popularity since Google introduced them in the early 2010s. According to a Google study, WebPs can be anywhere between 25% and 34% smaller than JPGs. What’s more, you can use both lossy and lossless compression on WebPs to get them down to even smaller sizes.

Something to keep in mind with WebPs is that they’re not universally accepted. As of writing this, WebPs aren’t accepted by iOS devices. However, the latest versions of all other browsers, Google or otherwise, will gladly display them.

As for how ImageKit helps with this, it’s simple really:

Image Kit image format settings

This ImageKit setting puts the responsibility on ImageKit to serve the best file format. (Source: ImageKit) (Large preview)

When this setting is configured, ImageKit automatically determines the best file format to deliver each of your files in. It takes into account what the original image format and content was along with whether or not the visitor’s device supports it.

JPGs, PNGs and GIFs will all be converted into WebPs when possible — say, if the visitor visits from Chrome (which accepts them). If it’s not possible — say, if the visitor visits from Safari (which doesn’t accept them) — ImageKit will convert to the best (i.e. smallest) format with the defined transformations. This might be a PNG or JPG.

Nykaa was able to capitalize on this image optimization strategy from ImageKit. Even though their website had already been designed using a mix of JPGs and PNGs and were stored in a number of places around the web, ImageKit took care of automating the image formats right from the original URLs.

3. Compress Images

Next, we need to talk about image compression. I’ve already referenced this a couple times, but it breaks down to two types:

Lossless

This form of compression is used on PNGs and GIFs. To compress the file, metadata is stripped out. This way, the integrity of the image remains intact, but the file shrinkage isn’t as substantial as you’d get with lossy compression.

Lossy

This form of compression is applied to JPGs and WebPs. To compress the file, some parts of the image are “lost”, which can give certain spots a granier appearance than the original image. In most cases, it’s barely noticeable unless you look closely at a side-by-side of the two images. But to your visitors, the degradation is easy to miss since there’s no original to compare against.

With lossy compression, you get to control what percentage of the file degrades. A safe range would be anything over 70% to 80%. ImageKit, by default, sets its optimization for 80% and it estimates that you can save at least 20% to 25% of your file size just from that. In reality, though, it’s probably more (we’re looking at upwards of 40% like in the Unsplash image example above):

ImageKit lossy compression settings

This ImageKit setting enables its users to decide how much lossy compression they want applied to their JPGs. (Source: ImageKit) (Large preview)

You can change this to whatever default you believe will maintain quality while giving you the image sizes that help your site load quickly.

Whether you use the default or your own optimization setting, remember to switch on the additional compression settings available under the Advanced tab.

ImageKit advanced optimization settings

ImageKit provides Advanced image optimization settings for JPGs and PNGs. (Source: ImageKit) (Large preview)

These three settings, in particular, will enable you to do as much compressing and as safely as possible.

The first setting “Save a Copy”, for instance, keeps your original images on the ImageKit server. That way, you have a copy of the image pre-compression without having to manage the burden of it on your own server.

The second setting “Preserve Image Metadata” enables you to apply lossless compression when feasible.

And the last setting “PNG Image Compression Mode” allows you to decide what level of lossless optimization you want to use on your PNGs: maximum, minimum or none.

When done, you’ll end up with results like this side-by-side comparison:

Comparison between compressed and original JPG from Unsplash

This side-by-side comparison of an Unsplash image from Luke Jeremiah shows a compressed file and an original JPG. (Source: Unsplash) (Large preview)

This is a JPG from Unsplash. Can you tell which is the original and which is the compressed and resized version from ImageKit?

The one on the left with the black trim is:

  • 1500×1005 px
  • 266 KB
  • Compressed at 95%

The one on the right with the white trim is:

  • 5444×3649 px
  • 2.5 MB
  • Original

It’s up to you to decide which of the ImageKit compression and optimization settings you’re most comfortable using and then configure accordingly.

4. Save to and Pull Images from External Server

There are two ways to run images through ImageKit.

The first is by uploading your images directly to its Media Library:

ImageKit Media Library

ImageKit allows users to store their images in its Media Library instead of their own servers. (Source: ImageKit) (Large preview)

The second is by integrating with your website or external storage service. We’ve actually already seen this part of ImageKit. It’s where you get your URL endpoints from so you can define your image parameters:

ImageKit integrations

ImageKit integrates with content management systems, third-party storage and Cloudinary. (Source: ImageKit) (Large preview)

Even with all of the optimizations above, you might still be having a hard time with image storage and maintenance — either because of how they affect your speed or how much storage you have to hold them.

For instance, if you store your images on your server, you’ll eventually be constrained for space (unless you have a monster-sized hosting account).

When you’re building massive e-commerce stores or business websites with thousands or even millions of images and corresponding image sizes, you can’t afford to be hosting those images on your own. Granted, there is a way to serve them more quickly to visitors (which I’ll explain in the next point), but why take on the burden and cost of additional storage if you don’t have to?

5. Add a CDN

A CDN is another essential optimization tool for large repositories of images. Think of it like a second server, only this one caches (copies) your website and serves them through data centers located significantly closer to your visitors around the world.

As a result, the time it takes to send your website and its thousands of product images from New York, New York to Bangladesh, India happens insanely fast.

With ImageKit, you get to enjoy the privilege of serving your images not just through its core processing servers, but through AWS CloudFront CDN (included in all plans) which has over 150 locations worldwide.

Sintra, a client of ImageKit, saw a big leap in performance after moving to ImageKit. With the ImageKit image CDN (which has delivery nodes all around the globe), it saw an 18% drop in page load times.

Wrapping Up

What’s especially nice about ImageKit is that it’s not just a preventative measure against slowdowns caused by images. You can use it to retroactively fix and improve mobile websites and PWAs, even if they already have millions of images on them. What’s more, the performance center makes it easy to keep an eye on your website’s images and identify opportunities for speed enhancements.

Plus, as you can see from the tips above, ImageKit has simplified a lot of the work you’d otherwise have to do, whether you’d handle it manually or configure it through a plugin.

With consumers and Google becoming pickier by the day about how quickly websites load on mobile, this is the kind of image optimization strategy you need. It’ll lighten your load while ensuring that any images added before or after ImageKit are optimized to the fullest. Even better, your clients will reap the benefits of more leads and greater conversions.

Smashing Editorial
(yk)

Source: Smashing Magazine, A Guide To Optimizing Images For Mobile

Collective #559

dreamt up by webguru in Uncategorized | Comments Off on Collective #559

C559_gridsome

Gridsome

Gridsome is a free and open source Vue.js-powered framework for building websites and apps that are fast by default.

Check it out




C559_webwide

Webwide

A free and inclusive discussion community for web designers, developers and makers.

Check it out



C559_TinaCMS

TinaCMS

In case you didn’t stumble upon it yet: Tina is an open-source site editing toolkit for React-based frameworks – Gatsby and Next.js

Check it out








C559_orientation

Image orientation on the web

In this article you will learn about the current status of image orientation on the web, how to correct orientation of images using Node.js, and how browsers will handle this in the future.

Read it



C559_zero

Zero

Zero is a small graphics app that uses JavaScript to replicate the functionality of a GPU and uses the terminal to display its rendered output via nodejs’ stdout.

Check it out


C559_cobeats

CoBeats

CoBeats helps you keep and organize all web things like bookmarks, screenshots, videos, files, images and more.

Check it out









Collective #559 was written by Pedro Botelho and published on Codrops.


Source: Codrops, Collective #559

Great Expectations: Using Story Principles To Anticipate What Your User Expects

dreamt up by webguru in Uncategorized | Comments Off on Great Expectations: Using Story Principles To Anticipate What Your User Expects

Great Expectations: Using Story Principles To Anticipate What Your User Expects

Great Expectations: Using Story Principles To Anticipate What Your User Expects

John Rhea



Whether it’s in a novel, the latest box office smash, or when Uncle Elmer mistook a potted cactus for a stress ball, we all love stories. There are stories we love, stories we hate, and stories we wish we’d never experienced. Most of the good stories share structure and principles that can help us create consistent website experiences. Experiences that speak to user expectations and guide them to engage with our sites in a way that benefits both of us.

In this article, we’ll pull out and discuss just a few examples of how thinking about your users’ stories can increase user engagement and satisfaction. We’ll look at deus ex machina, ensemble stories, consistency, and cognitive dissonance, all of which center on audience expectations and how your site is meeting those expectations or not.

We can define a story as the process of solving a problem. Heroes have an issue, and they set out on a quest to solve it. Sometimes that’s epic and expansive like the Lord of the Rings or Star Wars and sometimes it’s small and intimate such as Driving Miss Daisy or Rear Window. At its core, every story is about heroes who have a problem and what they do to solve it. So too are visits to a website.

The user is the hero, coming to your site because they have a problem. They need to buy a tchotchke, hire an agency or find the video game news they like. Your site can solve that problem and thus play an important role in the user’s story.

Deus Ex Machina

It’s a term meaning “god from the machine” that goes back to Greek plays — even though it’s Latin — when a large, movable scaffolding or “machine” would bring out an actor playing a god. In the context of story, it’s often used to describe something that comes out of nowhere to solve a problem. It’s like Zeus showing up at the end of a play and killing the villain. It’s not satisfying to the audience. They’ve watched the tension grow between the hero and the villain and feel cheated when Zeus releases the dramatic tension without solving that tension. They watched a journey that didn’t matter because the character they loved did not affect the ending.

The danger of deus ex machina is most visible in content marketing. You hook the audience with content that’s interesting and applicable but then bring your product/site/whatever in out of nowhere and drop the mic like you won a rap battle. The audience won’t believe your conclusion because you didn’t journey with them to find the solution.

If, however, the author integrates Zeus into the story from the beginning, Zeus will be part of the story and not a convenient plot device. Your solutions must honor the story that’s come before, the problem and the pain your users have experienced. You can then speak to how your product/site/whatever solves that problem and heals that pain.

State Farm recently launched a “Don’t Mess With My Discount!” campaign:

Kim comes in to talk to a State Farm rep who asks about a Drive Safe and Save discount. First, for the sake of the discount, Kim won’t speed up to make a meeting. Next, she makes herself and her child hold it till they can get home driving the speed limit. Last, in the midst of labor, she won’t let her partner speed up to get them to the hospital. (Don’t mess with a pregnant lady or her discount.) Lastly, it cuts back to Kim and the agent.

State Farm’s branding and their signature red color are strong presences in both bookend scenes with the State Farm representative. By the end, when they give you details about their “Drive Safe and Save” discount you know who State Farm is, how they can help you, and what you need to do to get the discount.


It’s not a funny story that’s a State Farm commercial in disguise, but a State Farm commercial that’s funny.

Throughout the ad, we know State Farm’s motivations and don’t feel duped into liking something whose only goal is to separate us from our money. They set the expectation of this story being an ad in the beginning and support that throughout.

Another Approach

Sometimes putting your name upfront in the piece might feel wrong or too self-serving. Another way to get at this is to acknowledge the user’s struggle, the pain the user or customer already feels. If your site doesn’t acknowledge that struggle, then your product/site/whatever seems detached from their reality, a deus ex machina. But if your content recognizes the struggle they’ve been through and how your site can solve their problem, the pitch for deeper engagement with your site will be a natural progression of the user’s story. It will be the answer they’ve been searching for all along.

Take this testimonial from Bizzabo:

Emily Fullmer, Director of Global Events for Greenbook said, We are now able to focus less on tedious operations, and more on creating a memorable and seamless experience for our attendees.

Bizzabo solved a real world problem for Greenbook. (Large preview)

It shows the user where Greenbook was, i.e. mired in tedious tasks, and how Bizzabo helped them get past tedium to do what Greenbook says they do best: make memorable experiences. Bizzabo doesn’t come out of the woodwork to say “I’m awesome” or solve a problem you never had. They have someone attesting to how Bizzabo solved a real problem that this real customer needed to be fixed. If you’re in the market to solve that problem too, Bizzabo might be the place to look.

Ensemble Stories

Some experiences, like some stories, aren’t about a single person. They’re about multiple people. If the story doesn’t give enough attention to each member, that person won’t seem important or like a necessary part of the story. If that person has a role in the ending, we feel cheated or think it’s a deus ex machina event. If any character is left out of a story, it should change the story. It’s the same way with websites. The user is the story’s hero, but she’s rarely the only character. If we ignore the other characters, they won’t feel needed or be interested in our websites.

Sometimes a decision involves multiple people because a single user doesn’t have the authority to decide. For instance, Drupalcon Seattle 2019 has a “Convince Your Boss” page. They showcase the benefits of the conference and provide materials to help you get your boss to agree to send you.

You could also offer a friends-and-family discount that rewards both the sharer and the sharee. (Yes, as of this moment, “sharee” is now a word.) Dropbox does this with their sharing program. If you share their service with someone else and they create an account, you get additional storage space.

Dropbox offers an additional 250 MB of space for every friend you get to join their service.

You get extra space, you get extra space, and you get extra space (when you invite a friend). (Large preview)

But you don’t have to be that explicit about targeting other audiences than the user themselves. In social networks and communities, the audience is both the user and their friends. The site won’t reach a critical mass if you don’t appeal to both. I believe Facebook beat MySpace early on by focusing on the connection between users and thus serving both the user and their friends. MySpace focused on individual expression. To put it another way, Facebook included the user’s friends in their audience while MySpace didn’t.

Serving Diametrically Opposed Heros

Many sites that run on ad revenue also have to think about multiple audiences, both the users they serve and the advertisers who want to reach those users. They are equally important in the story, even if their goals are sometimes at odds. If you push one of these audiences to the side, they’ll feel like they don’t matter. When all you care about is ad revenue, users will flee because you’re not speaking to their story any longer or giving them a good experience. If advertisers can’t get good access to the user then they won’t want to pay you for ads and revenue drops off.

Just about any small market newspaper website will show you what happens when you focus only on advertisers’ desires. Newspaper revenue streams have gone so low they have to push ads hard to stay alive. Take, for instance, the major newspaper from my home state of Delaware, the News Journal. The page skips and stutters as ad content loads. Click on any story and you’ll find a short article surrounded by block after block after block of ad content. Ads are paying the bills but with this kind of user experience, I fear it won’t be for long.

Let me be clear that advertisers and users do not have to be diametrically opposed, it’s just difficult to find a balance that pleases both. Sites often lean towards one or the other and risk tipping the scales too far either way. Including the desires of both audiences in your decisions will help you keep that precarious balance.

One way to do both is to have ads conform to the essence of your website, meaning the thing that makes your site different i.e. the “killer app” or sine qua non of your website. In this way, you get ads that conform to the reason the users are going to the site. Advertisers have to conform to the ad policy, but, if it really hits on the reason users are going to the site, advertisers should get much greater engagement.

On my own site, 8wordstories.com, ads are allowed, but they’re only allowed an image, eight words of copy, and a two-word call to action. Thus when users go to the site to get pithy stories, eight words in length, the advertisements will similarly be pithy and short.


Advertisers and users do not have to be diametrically opposed, it’s just difficult to find a balance that pleases both.

Consistency

The hero doesn’t train as a medieval knight for the first half of the story and then find herself in space for the second half. That drastic shift can make the audience turn on the story for dashing their expectations. They think you did a bait-and-switch, showing them the medieval story they wanted and then switching to a space story they didn’t want.

If you try to hook users with free pie, but you sell tubas, you will get lots of pie lovers and very few tuba lovers. Worse yet is to have the free pie contingent on buying a tuba. The thing they want comes with a commitment or price tag they don’t. This happens a lot with a free e-book when you have to create an account and fill out a lengthy form. For me, that price has often been too high.

Make sure the way you’re hooking the audience is consistent with what you want them to read, do, or buy. If you sell tubas offer a free tuba lesson or polishing cloth. This’ll ensure they want what you provide and they’ll think of you the next time they need to buy a tuba.

That said, it doesn’t mean you can’t offer free pie, but it shouldn’t get them in the door, it should push them over the edge.

Audible gives you a thirty-day free trial plus an audio book to keep even if you don’t stay past the trial. They’re giving you a taste of the product. When you say, “I want more.” You know where to get it.

While not offering a freebie, Dinnerly (and most of the other bazillion meal kit delivery companies) offers a big discount on your first few orders, encouraging new customers to try them out. This can be an especially good model for products or services that have fixed costs with enticing new customers.

Dinnerly offers a discount on each of your first three meal kits.

Hmmm… maybe they should offer free pie. (Large preview)

Cognitive Dissonance

There’s another danger concerning consistency, but this one’s more subtle. If you’re reading a medieval story and the author says the “trebuchet launched a rock straight and true, like a spaceship into orbit.” It might be an appropriate allusion for a modern audience, but it’s anachronistic in a medieval story, a cognitive dissonance. Something doesn’t quite make sense or goes against what they know to be true. In the same way, websites that break the flow of their content can alienate their audience without even meaning to (such as statistics that seem unbelievable or are so specific anyone could achieve them).

112% of people reading this article are physically attractive.

(Here’s lookin’ at you, reader.)

This article is the number one choice by physicians in Ohio who drive Yugos.

(Among other questions, why would a European car driving, Ohioan Doctor read a web user experience article?)

These “statistics” break the flow of the website because they make the user stop and wonder about the website’s reputability. Any time a user is pulled out of the flow of a website, they must decide whether to continue with the website or go watch cat videos.

Recently, I reviewed proposals for a website build at my day job. The developers listed in the proposal gave me pause. One with the title “Lead Senior Developer” had seven years of experience. That seemed low for a “lead, senior” developer, but possible. The next guy was just a “web developer” but had twenty years of experience. Even if that’s all correct, their juxtaposition made them look ridiculous. That cognitive dissonance pulled me out of the flow of the proposal and made me question the firm’s abilities.

Similarly poor quality photos, pixelated graphics, unrelated images, tpyos, mispelllings, weird bolding and anything else that sticks out potato can cause cognitive dissonance and tank a proposal or website (or article). The more often you break the spell of the site, the harder it will be for clients/users to believe you/your product/site/thing are as good as you say. Those cat videos will win every time because they always meet the “lolz” expectation.

Conclusion

Users have many expectations when they come to your site. Placing your users in the context of a story helps you understand those expectations and their motivations. You’ll see what they want and expect, but also what they need. Once you know their needs, you can meet those needs. And, if you’ll pardon my sense of humor, you can both …live happily ever after.

Smashing Editorial
(cct, ra, yk, il)

Source: Smashing Magazine, Great Expectations: Using Story Principles To Anticipate What Your User Expects