Website

Pull

Synopsis

Director loads one web page in its headless Chromium, from where it runs, as a visitor's browser loads it: its scripts run, and its images, styles, and fonts come. At every check it writes the result itself as an ECS document: the page's timings, the Web Vitals one load can measure, its requests and console counted, and the state the page is in — up or down, since when, and after how many checks. Nothing is installed on the site.

The Website is one of six service check devices, each checking one service: a Website, an HTTP Check, a TCP Check, a DNS Check, an ICMP Check, and a TLS Check.

One device, one page. A device loads one URL, so every page checked has a row of its own in the fleet list, and the device's connection state says whether that page is up.

The results are the device's own. Every check writes a result, up or down, with no dataset assigned. Datasets are for what comes beside the results — what a page load finds (its console, its requests, the sitespeed.io coach's advice, its filmstrip, its screenshots), the scripts the website runs step by step, and the route to its host. See Service Check Datasets.

One Director owns each device.

note

The service check devices need Director 2.2.3 or later.

Schema

- id: <numeric>
  name: <string>
  description: <string>
  type: website
  tags: <string[]>
  status: <boolean>
  properties:
    url: <string>
    expected_status: <numeric[]>
    tls:
      insecure_skip_verify: <boolean>
    username: <string>
    password: <string>
    headers: <map>
    hmac:
      header: <string>
      key: <string>
      type: <string>
      prefix: <string>
    interval: <numeric>
    timeout: <numeric>
    viewport:
      width: <numeric>
      height: <numeric>
    jpeg_quality: <numeric>
    thumbnail_width: <numeric>
    filmstrip_frames: <numeric>
    max_record_bytes: <numeric>
    change_threshold: <numeric>
  pipelines: <pipeline[]>
  definitions: <definition[]>

Configuration

The following fields are used to define the device:

Device

Field	Required	Default	Description
`id`	Y	-	Unique numeric identifier
`name`	Y	-	Device name
`description`	N	-	Optional description
`type`	Y	-	Must be `website`
`tags`	N	-	Optional tags
`status`	N	`true`	Enable/disable the device

Page

Field	Required	Default	Description
`url`	Y	-	`http://` or `https://`, with a host that is not a wildcard. A URL that carries `user:password@` is refused. A credential-shaped query parameter is masked in every record
`expected_status`	N	any 2xx	The status codes of the page's main document, after its redirects, that mean up
`tls.insecure_skip_verify`	N	`false`	`true` loads a page whose certificate the browser does not trust, for this device's pages alone, its synthetic runs' among them

A browser asks with GET, follows every redirect, reads every body, and verifies a certificate against its own trust store. An HTTP Check's method, body_match, follow_redirects, and other tls.* settings are said once in the device's log and ignored.

Credentials for Synthetic Runs

A website's page loads send no credentials. These are what its Website Synthetic Check datasets' pages send, and a device with no such dataset is told, once, that they are ignored.

Field	Required	Default	Description
`username`, `password`	N	-	Sent as basic authentication, when both are set. Plain text, `${ENV_VAR}`, or `$secret{...}`
`headers`	N	-	Sent when no basic authentication is: a map of header names to values, each plain text, `${ENV_VAR}`, or `$secret{...}`. `Host` is not accepted, as it always follows the URL
`hmac.header`	N	-	Sent when neither of those is: the name of a header carrying an HMAC
`hmac.key`	N	-	The HMAC's key. Plain text, `${ENV_VAR}`, or `$secret{...}`
`hmac.type`	N	-	`sha1`, `sha256`, or `sha512`
`hmac.prefix`	N	-	Written before the HMAC, in the same header

Only one of them is sent, the first that is set: basic authentication, else the headers, else the HMAC header. One set beside another that goes first is said in the device's log, and not sent. The HMAC header holds the HMAC of an empty message under hmac.key, in hexadecimal, after hmac.prefix: it signs no message, so it is the same value on every request, a token made from the key.

They go only to the device's own sites. A synthetic run sends them with its page's requests to the device's URL and the pages its script visits (vm.visit), each matched exactly by its scheme, host, and port. Every other request gets none: a third party the page loads (fonts, CDNs, analytics), a link to another site, or a redirect's hop to another origin. On the sites that do get them, basic authentication is sent up front, without waiting for a 401.

Each is resolved at every run, so a password rotated in a vault is sent from the next run on. The API withholds the password, every header's value, and hmac.key from its device reads.

Pipeline

The device takes at most one pre-processing pipeline of its own: the device's pipeline picker in the console, pipelines: in a configuration file. Its results go through it, and through none when the device has none. A dataset's records go through the pipeline the dataset names (pipeline_name), and through the device's when it names none.

Datasets

The device's definitions are its datasets: website_console_collector, website_har_collector, website_coach_collector, website_filmstrip_collector, website_screenshot_collector, website_synthetic_collector, and traceroute_collector. In the console they are the datasets and profiles assigned to the device. For their fields, see Service Check Datasets.

With no dataset assigned, the device still writes its results.

Advanced Configuration

Field	Required	Default	Description
`interval`	N	`60`	Seconds between page loads: a result every interval. At least `30`, since every check loads the page
`timeout`	N	`30`	Seconds from the navigation to the page's load event, from `1` to `120`
`viewport.width`, `viewport.height`	N	`1280` × `800`	CSS pixels, at one device pixel each: `320`–`3840` wide, `240`–`2160` high. Also a full screenshot's size
`jpeg_quality`	N	`60`	Every picture's JPEG quality, from `10` to `95`
`thumbnail_width`	N	`320`	The thumbnail's and the filmstrip frames' width, in pixels, from `64` to `1280`, and no wider than the viewport
`filmstrip_frames`	N	`8`	Frames a filmstrip keeps, from `2` to `30`
`max_record_bytes`	N	`262144` (256 KiB)	The largest record a picture goes into, from `16384` to `4194304` (4 MiB). A picture past it is scaled down, or left out, never cut
`change_threshold`	N	`12`	Bits of 256 the screenshot's perceptual hash moves for a visible change, from `1` to `128`

A configuration error stops the device with Configuration error, and the message names the property, never its value.

The Director's Browser

One browser serves every website of a Director. It starts at the first check, starts again at the next one when it went away (a crash, a kill), and closes when the last website stops, so a Director with no website runs no browser. It is found:

at browser.path in the Director's vmetric.yml, when set: an executable, a path that is not absolute being relative to the Director's own directory;
else in the chrome folder of the Director's package directory, package/chrome/chrome (package\chrome\chrome.exe on Windows), where the Director's package archive carries Chromium.

It is looked for at every check, so a browser put in place, or a setting changed, is taken at the next one. A Director with no browser there fails every check with Initialization error, and the message names where it looked (no browser at /opt/director/package/chrome/chrome: ...). No record is written, and no browser is ever downloaded.

Headless, with site isolation. The browser runs headless, and without its sandbox when the Director runs as root, which Chromium refuses to sandbox. A frame of another site runs in a process of its own.
browser.leakless: true in vmetric.yml starts the browser through go-rod's leakless guard, a process that kills the browser however the Director goes. It is off by default. The guard is an executable written into the system's temporary directory, which a virus scanner may stop; a start that has not answered within 30 seconds fails the check, and the next check starts the browser again. Windows on Arm has no guard.
Proxy. The browser reaches the page through the proxy the Director's environment names (HTTPS_PROXY, HTTP_PROXY, and NO_PROXY for the hosts that bypass it), without the credentials a proxy URL may carry. With none named, the browser finds the system's proxy itself; on Windows that costs a check about a second, which naming the proxy avoids.

Every check loads the page in a browser context of its own, closed with the check: no cookie, cache entry, connection, or service worker of one check is another's, so each one measures its names resolved, its connections made, and its TLS handshakes.

The page sees the browser as it is: the user agent the Director's browser gives, with its own version and system and HeadlessChrome said as Chrome; the browser's own client hints; and Accept-Language: en. A page can still tell it is driven: navigator.webdriver is true.

How a Check Works

A website loads its page once a reading, and its results and every dataset due then write what they read of that load: those due within a second of each other share one load rather than load the page twice. After its load event, the page is given until no request has been in flight for half a second, three seconds at most, so what runs late (a lazy image, the largest paint, a long task) is counted.

The check is up when the page's navigation committed, its main document answered an expected status, and its load event came within the timeout. Otherwise it is down, and error.type says why:

`error.type`	When
`dns`	The host does not resolve
`certificate`	The browser does not trust the certificate
`connection`	The connection failed
`redirect`	The redirects failed
`timeout`	No answer, or no load event, within the timeout
`status`	The main document answered a status not expected
`navigation`	The browser's other network errors
`crashed`	The page crashed

A failed load is never a page that loaded with errors. A page that loads is up even when a request of it got no answer, a script wrote errors to the console, or an exception went uncaught: its result counts them, and the Website Console and Website HAR datasets say what they were. What a page that is down came to is still read, its screenshot among it: the browser's error page, or the page as far as it got.

A check that could not be made — no browser, a browser that went away under the check — writes no result and is not counted in the state.

Results and State

Every check writes one result, with event.dataset website and definition id 12154, to the metric table:

url.* (the page asked for); http.response.status_code, mime_type, and http.version (the main document's, after its redirects); destination.address, ip, port.
service_check.browser.loaded, final_url, navigation_error (the browser's net error), redirects.
service_check.browser.timings.redirect, dns, connect, tls_handshake, first_byte, download (the main document's phases it went through), and dom_interactive, dom_content_loaded, load, first_contentful_paint, largest_contentful_paint (from the navigation's start).
service_check.browser.timings.total_blocking_time, cumulative_layout_shift, largest_contentful_paint_element.
service_check.browser.requests.total, failed, error_responses, transferred.bytes; console.errors, warnings, exceptions.

The timings are the Navigation Timing phases of the main document and the Web Vitals one load can measure. Interaction to Next Paint is left out: it needs a user's input, which one page load has none of. A phase the load did not go through, and a Web Vital the browser does not report, are left out, never written as zero.

Every result carries the state the page is in, under state.*. For the state's fields, how to alert on it, and the fields every record carries, see Results and State.

Connection States

State	When
Connected	The page is up
Connection error	It does not answer, its name does not resolve, or the browser could not load it
Certificate error	The browser does not trust its certificate
Error	It answered a status not expected, or it crashed
Configuration error	A configuration that does not validate
Initialization error	No browser where the Director looks for one, a browser that does not start, went away under the check, or sent an event the Director cannot read

Console

In the console, go to Fleet Management > Devices > Website and click Add new device. The wizard's steps are:

General settings, as on every device, with the device's pre-processing pipeline.
Check settings: under Page, the URL and Expected status codes (Optional), with a note on the browser the Director loads the page in; for an https:// URL, the TLS settings; then Credentials for synthetic runs: Username (Optional), Password (Optional), the headers, and HMAC header (Optional), HMAC key (Optional), HMAC algorithm (Optional), and HMAC prefix (Optional). With more than one of them set, the wizard warns that only one is sent.
Advanced settings: Interval and Timeout; then, under Viewport and pictures, Viewport width, Viewport height, JPEG quality, Thumbnail width, Filmstrip frames, Change threshold, and Maximum record size.
Data collection: the datasets or profile the device writes. A device receives datasets or a profile, never both.
Review and create.

tip

Screenshots and filmstrip frames are JPEG images carried as base64 in their records. Route the Website Screenshot and Website Filmstrip datasets to a target that takes large records, such as blob storage, and leave the SIEM the results, the console, and the requests.

Simulation

The Website is not simulated. What it writes is what its page load finds, with the state the page is in, which one generic sample cannot stand in for.

Examples

Page with Its Datasets

Loading a shop's home page every minute, writing its console and requests, and a filmstrip every five minutes...

devices:
  - id: 512300184
    name: shop-browser
    type: website
    status: true
    properties:
      url: "https://shop.example.com/"
      expected_status: [200]
    definitions:
      - name: website_console_collector
      - name: website_har_collector
      - name: website_filmstrip_collector
        properties:
          interval: 300
      - name: website_screenshot_collector

Signing in every fifteen minutes, with the script kept in the Director's environment because it types a password, and basic authentication from a vault sent to the shop's own site alone...

devices:
  - id: 512300187
    name: shop-login
    type: website
    status: true
    properties:
      url: "https://shop.example.com/"
      username: "svc-monitor"
      password: "$secret{store=corp-vault,ref=shop/monitor}"
    definitions:
      - name: website_synthetic_collector
        properties:
          transaction: "${SHOP_LOGIN_SCRIPT}"
          cron: "*/15 * * * *"

Synopsis​

Schema​

Configuration​

Device​

Page​

Credentials for Synthetic Runs​

Pipeline​

Datasets​

Advanced Configuration​

The Director's Browser​

How a Check Works​

Results and State​

Connection States​

Console​

Simulation​

Examples​

Page with Its Datasets​

Synthetic Sign-in​