I thought it would be possible to closely monitor the appearance using Google Chrome's headless mode, so I tried it.

Hello.
I'm Mandai, in charge of Wild on the development team.

Beyond Co., Ltd. is an MSP, so we perform various monitoring tasks according to customer orders

Among these is external monitoring, which involves accessing the site in the same (or similar) way that a user would access the site and checking whether the page is displayed correctly

There are various tools available to access a site and check whether data is being collected normally, and it is also possible to do it yourself using curl, etc., but with these methods, the cause can only be discovered if the person who receives an alert from the various monitoring tools accesses the site

That's fine, but this time I'd like to explore the possibility that by using Google Chrome's headless mode, it's possible to obtain detailed information via DevTools about which content is taking a long time to retrieve, which may make it possible to determine the cause without looking at the site

Preparation

First, gather everything you need to try it out.
There will also be a sample that works by copying and pasting it later.

 

Google Chrome

First, install Google Chrome (version 59 or later) running in headless mode.
This version of Google Chrome is already stable, so you can install it using the installer as usual.
If automatic updates are enabled, the update should already be installed.

 

Node.js

This time I used the latest version, 8.0.0, but I think it would be fine even if it was a fairly old version

The following modules are installed with npm:

• chrome-launcher
• chrome-remote-interface

chrome-launcher is used to launch Google Chrome from Node.js.
chrome-remote-interface is used to access Google Chrome DevTools from Node.js.

npm install chrome-launcher chrome-remote-interface

 

The sample script includes a process that starts Google Chrome every time the application is started and stops it when the application is closed, but if you are using it seriously, it may be fine to leave Google Chrome running (although it seems like memory usage will continue to increase, so regular restarts may be necessary)

Once you have these two installed, you're ready to go

For basic information on using Google Chrome's headless mode, please refer to "
Let's play around with headless mode, now standard in Google Chrome 59." For basic information on using Google Chrome's headless mode from Node.js, please refer to " Using headless Google Chrome from Node.js.

 

Sample script

We have prepared the following samples, so please feel free to use them!

const ChromeLauncher = require('chrome-launcher'); const CDP = require('chrome-remote-interface'); function launchChrome(headless = true) { ChromeLauncher.launch({ port: 9222, autoSelectChrome: true, chromeFlags: [ '--disable-gpu', headless ? '--headless' : '', '--no-sandbox', ], }).then(launcher => { CDP(protocol => { setTimeout(() => { protocol.close(); launcher.kill(); process.exit(); }, 10000); const {Page, Network} = protocol; Promise.all([ Page.enable(), Network.enable(), ]).then(() => { Page.navigate({url : 'https://beyondjapan.com/blog/2017/07/headless-chrome-networks'}); Network.responseReceived(res => { console.log(r) }) }) }) }) } launchChrome();

 

It doesn't terminate automatically, so I use setTimeout to terminate it after 10 seconds.
If there is a lot of communication on one page, there will be a lot of output, but if you extract one

{ requestId: '30371.7375', frameId: '30371.1', loaderId: '30371.190', timestamp: 1197407.412915, type: 'Document', response: { url: 'https://beyondjapan.com/', status: 200, statusText: 'OK', headers: { Date: 'Tue, 18 Jul 2017 04:16:02 GMT', 'Transfer-Encoding': 'chunked', Server: 'nginx', Connection: 'keep-alive', Link: '<https://beyondjapan.com/cms-json/> ; rel="https://api.w.org/"', Vary: 'Accept-Encoding', 'Content-Type': 'text/html; charset=UTF-8' }, mimeType: 'text/html', connectionReused: false, connectionId: 5394, remoteIPAddress: '122.218.102.93', remotePort: 80, fromDiskCache: false, fromServiceWorker: false, encodedDataLength: 253, timing: { requestTime: 1197406.917743, proxyStart: -1, proxyEnd: -1, dnsStart: 0.944999977946281, dnsEnd: 2.30200006626546, connectStart: 2.30200006626546, connectEnd: 7.89500004611909, sslStart: -1, sslEnd: -1, workerStart: -1, workerReady: -1, sendStart: 7.98300001770258, sendEnd: 8.11700010672212, pushStart: 0, pushEnd: 0, receiveHeadersEnd: 494.426999939606 }, protocol: 'http/1.1', securityState: 'neutral' } }

 

It seems that most of the information available in the Network tab of DevTools is available.
All timestamps appear to be in milliseconds.

It also seems to be able to do everything DevTools can do, such as deleting or adding cookies (which may be useful for monitoring sites that use sessions), clearing the cache, rewriting the user agent, etc. It
would also be fun to create a UI that displays data obtained using something like D3.js!

 

summary

What do you think of external monitoring using Google Chrome's headless mode?

I think it would be less expensive to use a monitoring tool, but since it can capture the responses of all the content being loaded, it seems like it could be useful as it would allow you to instantly determine if there is something wrong with the CDN or if the response from the API is extremely slow

That's it.