`lib-prov` 库参考

原文: https://finproducts.atlassian.net/wiki/spaces/JIDD3/pages/34793750747/FIN+5.3.1+lib-prov 中文整理基于 FIN 5.3.1

导读： 供应管理

以下为 API 参考，保留英文以便对照源码；目录已列出（英文标题便于检索）；概述已尽量中译。

目录（导读）

lib
meta
docs
tags
apps
funcs
views
def
supertypes
Overview
Setup
FileRepo
Prov Manifest
- provBuild
- provPatch
- provOverlay
- provNode
- provImage
Creating Prov Manifests
Provc
- Upload Action
- Stub Nodes
Deployment
- Patch Deployments
provDownloadBuild(ver, opts: {})
provDownloadPatches(build)
provNewImage(arg, dis, tags: {})
provSkySparkImage(version, manifests: null)
- usage
- tags
- subtypes
- src

概述（中译）

供应管理

API 正文（英文）

docs

Overview • Setup • FileRepo • Prov Manifest • Creating Prov Manifests • Provc • Deployment

tags


provBuild	The manifest contains all files for a SkySpark build
provImage	The manifest is an image and can be used to deploy files to nodes in the cluster
provManifest	A provisioning manifest
provNode	The manifest contains files which should be applied to a specific node in the cluster
provOverlay	The manifest contains files which should be applied after the SkySpark build and patches
provPatch	The manifest contains patches for a SkySpark build
provPlatform	The manifest contains files which should be applied to a specific platform as determined by the `hostModel` tag

apps


prov	Views used for provisioning software over the cluster

funcs


provDownloadBuild	Download a SkySpark build from skyfoundry.com and create or update a provBuild manifest for it
provDownloadPatches	Download all the current patches for a SkySpark build from skyfoundry.com and create or update a provPatch manifest for it
provNewImage	Create a provImage from an ordered list of manifest recs
provSkySparkImage	Create a provImage for the given SkySpark build number

views


prov	Cluster provisioning tool
provManifests	Manage provisioning manifests

app:prov

Views used for provisioning software over the cluster

meta


def	app:prov
disKey	ui::prov
icon	rocket
is	app
lib	lib:prov

supertypes


feature	Feature namespace of definitions formatted as `feature:name`
app	UI application grouping of views

lib prov

Overview

The provExt is used to upgrade and deploy files to SkySpark nodes connected in a cluster. One or more nodes in the cluster enable the provExt and are elevated to the status of Provisor and are then able to use the various tools provided in the ext to manage deployments on SkySpark nodes reachable in the cluster. The rest of this guide describes the core concepts and capabilities of the provExt.

To use the provExt both the Provisor node and the node(s) you wish to provision must be running at least SkySpark-3.0.22.

Setup

In order for the provExt to manage a set of SkySpark deployments, nodes wishing to participate in provisioning MUST adhere to the following requirements for deploying SkySpark.

Choose a directory where you wish to install SkySpark. We will call this directory the working directory. In the examples below, I have chose /opt/skyspark/ as the working directory, but you may choose any location that makes sense for your hosts.

The provExt "owns/manages" all contents of the working directory. To bootstrap provisioning you should structure your working directory to look like this (I am assuming you are bootstrapping provisioning with SkySpark-3.0.22):

/opt/skyspark/
             /skyspark-3.0.22/  # unzip skyspark-3.0.22.zip
             /var/              # var/ must be outside of skyspark deployment
             skyspark.sh        # or skyspark.bat on windows

NOTE: the var/ directory MUST be a direct child of the working directory. It cannot be inside your SkySpark deployment.

The contents of the skyspark.sh file should be

#!/bin/bash
cd /opt/skyspark/skyspark-3.0.22
chmod +x ./bin/*
./bin/skyspark "$@" –var /opt/skyspark/var/

If you are running Windows, then the contents of the skyspark.bat file should be:

@echo off
call "C:\opt\skyspark\skyspark-3.0.22\bin\skyspark.bat" %* -var "file:/C:/opt/skyspark/var/"

You should then configure the service manager for your OS/host to start SkySpark by calling skyspark.sh|bat script in your working directory. You should also configure your service manager to restart SkySpark automatically when the process terminates.

See the Deployment for details on what happens to the working directory during and after a deployment.

FileRepo

A necessary feature of any provisioning solution is the ability to move files from one node to another. Therefore, it is beneficial to have a database optimized for storing files. The "FileRepo" is a backing technology included with SkySpark for just this purpose. The FileRepo is mostly a behind-the-scenes actor when it comes to provisioning. Nevertheless, it is helpful to understand a little bit about how it works so that when we discuss provisioning manifests, their function makes more sense.

The FileRepo does not store files by name or path. Instead, we compute the SHA-256 hash of the file contents and store the file by its hash on disk. SHA-256 is a strong cryptographic hash function, which essentially guarantees that files with different content will yield different, unique hash values.

You can think of the FileRepo as a function that takes a file as input and returns a SHA-256 hash as output.

fileRepo(`etc/sys/ext2mime.props`) => c85123ccf30bc8b17dbd6c5aeae040eca5ef3a9931f7cad703da24ec102c98d9

All files that need to move across the cluster are first stored in the file repo. This yields a few benefits:

Between builds of SkySpark, there are many files which are basically static and do not change (e.g. bin/fan/skyspark or etc/sys/ext2mime.props). When we store them in the FileRepo, they will compute to the same file hash, and therefore we only have to save one copy instead of multiple.
We can reduce the amount of file transfer needed during provisioning. Before sending a file to a remote node, we can ask that node if it already has a file with a given hash in its local FileRepo. If it does, we don't need to transfer the file. Note - every node has a local FileRepo which serves as a cache of files that have been deployed on it.

To summarize, the FileRepo is a file storage database that stores files by their SHA-256 hash. We will refer to this hash as the fileHash.

Prov Manifest

A "Prov Manifest" is a special file which associates a fileHash with a physical location on disk. This information is stored as a Zinc Grid. Each row of the grid will have a fileHash and path tag. During deployment, the provExt will install the file with the given fileHash to the location specified by the path. If the path is null, then the provExt will assume that file is itself a Prov Manifest and recursively deploy its contents.

A Prov Manifest is the fundamental unit of deployment for the provExt. The order of files in a Prov Manifest is significant. During deployment, the files are installed in the order specified in the grid. If a row specifies a nested manifest (it is missing the path tag), then that manifest is "injected" at the current location and those files are installed before continuing.

The contents of a Prov Manifest might look like this:

path                      fileHash
----                      --------
etc/sys/ext2mime.props    c85123ccf30bc8b17dbd6c5aeae040eca5ef3a9931f7cad703da24ec102c98d9
etc/sys/units.txt         2e2bd35f196d6e4367c9faa4ac616d9809af2c54beca2b758bc313567a48b9d4
...

A Prov Manifest is itself stored in the FileRepo and has a unique fileHash. However, a SHA-256 hash is not a very friendly way of identifying a Prov Manifest. Whenever a Prov Manifest is generated, a provManifest folio record is also created that adds helpful metadata about the Prov Manifest so that they are easier to find and identify. At a minimum a provManifest will have these tags:

dis: A human-friendly display name describing the manifest.
provManifest: marker identifying the rec as metadata about a Prov Manifest
fileHash: the file hash of the Prov Manifest in the File Repo

There are several types of Prov Manifests that are further identified by the presence of one (and only one) of these marker tags:

provBuild

The provBuild tag indicates that the contents of the Prov Manifest are a full SkySpark build (e.g. 3.0.22). The manifest must contain the un-zipped contents of SkySpark build - not the zip itself.

Every provBuild manifest rec must also define a version tag so that tools can identify what version of SkySpark the build is for.

provPatch

The provPatch tag indicates that the contents of the Prov Manifest are the official, released patches from SkyFoundry for a SkySpark build. The provPatch rec must also define a version tag so that tools know which version of SkySpark the patches are for.

provOverlay

The provOverlay tag indicates that the contents of the Prov Manifest should be applied after the contents of a SkySpark build. This allows you maximum flexibility to "overlay" files on top of a standard SkySpark build thereby either replacing or adding to the build. This type of manifest would be the standard way to deploy custom exts, or make modifications to configuration files that are part of a build (e.g. etc/sys/config.props).

In order to create a provOverlay you will need to use the Provc tool described below.

provNode

The provNode tag indicates that the contents of the Prov Manifest are for a specific node in the cluster. This node is identified by the presence of a nodeRef tag on the rec. The canonical use-case for a provNode manifest is deployment of licenses for a particular node.

See the section on Deployment for details about how node manifests are handled during deployment.

provImage

The provImage tag indicates that the contents of the Prov Manifest is an ordered list of other provManifest recs to deploy. One useful property of all provManifest recs is that when they are ordered in a Grid, that Grid becomes a valid Prov Manifest. Each entry is missing the path tag; therefore a provImage is simply a Prov Manifest that consists of only nested Prov Manifests!

It is important to understand that a provImage represents an immutable, snapshot of the constituent provManfest recs at the time it was created. If those recs change after the provImage is created, it will not change the state of the image itself. You would need to recreate the provImage to pick up the changes.

Here is an example of the the Prov Manifest for a provImage that deploys the SkySpark-3.0.22 build and then its patches. Note - I am showing the contents in Trio format for readability, but all Prov Manifests are stored as Zinc.

id:@25431838-91327ae4 "SkySpark-3.0.22"
dis:"SkySpark-3.0.22"
fileHash:bda4a9cb75be04c8298fa88efa1fcf02d8ff04d7d83b997db113de429a5dcaca
provBuild
provManifest
version:"3.0.22"
---
id:@25431839-a558d213 "SkySpark-3.0.22 Patches"
dis:"SkySpark-3.0.22 Patches"
fileHash:"00e76f7b7a501a6a9a8ec8c9ee2def9320391b801c651a3a483bcd68c1cbc7d3"
provManifest
provPatch
version:"3.0.22"

The "Deploy" action of the Provisioning View will only allow you to deploy a provImage.

Creating Prov Manifests

Up until this point we have described the various Prov Manifests types, but we have not discussed how they are created in the first place. The section describes the various methods for creating and updating Prov Manifests.

If all you want to do is deploy standard SkySpark builds and patches, then you can use one of the built-in Axon funcs:

provDownloadBuild() - Download a SkySpark build from skyfoundry.com and save it as a provBuild manifest.
provDownloadPatches() - Download all the patches for a SkySpark build from skyfoundry.com and save them in a provPatch manifest.
provSkySparkImage() - Convenience to create the SkySpark build and patch manifests for a particular build, and then create an image all in one step.
provNewImage() - Create a provImage from a list of manifests.

If you want to deploy custom exts, files, or make modifications to files that are part of a SkySpark installation (e.g. etc/sys/config.props or etc/sys/units.txt), then you will need to create a provOverlay. There is a special tool called Provc that enables you to easily create overlays and upload them to SkySpark.

Provc

Provc (provisioning compiler) is a command-line tool that you can use to create custom manifests and upload them to SkySpark. Once uploaded to SkySpark, you can create images that contain those manifests using the Axon funcs mentioned above.

To run Provc from the command-line run:

$ ./bin/fan tools::Provc -help

Usage:
  provc <action> [options]

Available Actions:
  stubNodes    Stub node manifest definitions and staging directories
  upload       Upload provisioning manifests and files

Let's look at those in more detail.

Upload Action

The upload action is used to create/update a provManifest in a project running the provExt. If you need to create an overlay, then you will need to use this command.

There are two steps you need to do before you can upload a manifest:

Stage your overlay files
Create a trio file entry describing the manifest.

As a simple example, let's say you have two custom files you want to include with your SkySpark deployments:

A custom ext your wrote - acmeAwesomeExt.pod
A custom etc/sys/units.txt that contains additional units you need for your apps.

The first step is to stage these files on your file system in a directory structure that mirrors where they will be deployed in SkySpark. In the example below, we have staged the files under /manifests/myOverlay/

/manifests/myOverlay/
                    /lib/fan/acmeAwesomeExt.pod
                    /etc/sys/units.txt

Now we need to create or add an entry to a Trio file that describes this overlay manifest. In this example, we create a overlay.trio file with this contents

dis: "Acme Overlay"
provManifest
provOverlay
version: "1.0"
dir: `/manifests/myOverlay/`
---

Each rec in this trio file can have the following tags

dis: A display name describing the manifest
provManifest (required): The rec must have this tag to be a valid manifest
dir (required): This tells Provc what directory tree to scan for files to upload as part of this manifest.
<manifest type>: It should also be tagged with the appropriate manifest type. In this case we are creating a provOverlay
version (optional): You can tag your manifest with a version if you want to keep track of versioned changes to the manifest.
stackhub (optional): A Str[] list of StackHub pods to include in the manifest. If you specify just the name, then latest version is always retrieved. You can specify a specify version as <pod>-<version>. For example:
```
stackhub: ["sfxLbnlExt", "sfxCalculusExt-1.4.0"]
```

Now you can upload this manifest. In the example below I am uploading the files to the SkySpark instance running on host skysparkhost, and the manifest is being created in the myProv project. NOTE: you must use the api uri for your project.

$ ./bin/fan tools::Provc upload -uri http://skysparkhost/api/myProv overlay.trio

After this command completes, all files in the stage directory will be uploaded and stored in the File Repo of SkySpark and a provManifest rec will be created. If a provManifest rec already exists in the project and it matches the marker tags and optional version, then the existing rec will be updated instead.

Now that the manifest exists in your project, you can use one of the manifest creation funcs to create an image containing this manifest. Remember that manifests are applied in order so you will want to create an image where this manifest is applied after the standard SkySpark build. The provSkySparkImage() func can be used to do this.

Note: you can have multiple manifests defined in the same trio file. When you run the upload action they will all be uploaded to SkySpark.

Stub Nodes

If you want to do node-specific deployments, then you will need to create a nodeManifest for the node(s) you want to manage. This can be tedious if you have even a handful of nodes in your environment. The stubNodes action can be used to help in this cases.

When you run the stubNodes action, it will do the following:

Connect to your provisioning project and learn all the nodes that can be provisioned from that project.
Create staging directories for each node at a location you specify
Create a Trio file stubbing the provNode manifest for each node. This file can then be used as input to the upload action to upload and generate manifests for these nodes.
$ ./bin/fan tools::Provc stubNodes -uri http://skysparkhost/api/myProv -outDir ./provNodes/ nodes.trio

In this example, suppose that the myProv project is aware of two nodes, node1 and node2. Then the following will be stubbed:

./provNodes/
           /nodes.trio
           /node1/
                 /var/lic/
           /node2/
                 /var/lic/

And the contents of ./provNodes/nodes.trio will look something like this

id:@2540ad3e-2d512fcb "node1"
dis:"node1"
dir:`provNodes/node1/`
nodeRef:@n:node1
provManifest
provNode
---
id:@2540ad3e-d180120a "node2"
dis:"node2"
dir:`provNodes/node2/`
nodeRef:@n:node2
provManifest
provNode
---

You can now add node-specific files (e.g. licenses) into the various staging directories and then use the upload action to upload them.

NOTE: The provisioning engine currently only supports deploying license files. Another other files staged under var/ will result in a deployment error.

Deployment

When you use the Provisioning App to deploy an image to one or more nodes, the following high-level steps occur:

First, we apply all the manifests in the image, in order to produce a "base" image.
Then, for each node, we check if a node manifest exists for that node. If so, we create a node-specific image with the node manifest applied. This means the files in a node manifest have the highest priority for deployment.
Based on the final node image, we transfer any files the remote node is missing so that they are available in its local File Repo.
We send the final node image to the remote node. The remote node runs a local deployment to stage the image files. The node will terminate itself, which should force the Operating System service manager to restart SkySpark using the new deployment.

As discussed in the Setup section, the provExt owns/manages all files in the working directory. During a deployment, the provExt will create a new SkySpark directory with a unique name and stage the files in that directory. It will then re-write the skyspark.sh|bat file to boot SkySpark from this new location. The provExt does not currently remove old deployments as part of provisioning.

Patch Deployments

If you use the "Patch" action of the Provisioning App, then only the current SkySpark patches will be deployed to the remote node. In this case, we do not need to create a new deployment directory. We will patch SkySpark "in-place" (though a restart will still be required).

def

func:provDownloadBuild

provDownloadBuild(ver, opts: {})

Download a SkySpark build from skyfoundry.com and create or update a provBuild manifest for it. This also downloads all current patches for the build and creates or updates a provPatch for it.

provDownloadBuild("3.0.22")

The opts parameters supports the following options:

excludePaths😦Uri[,]) List of relative path URIs for files to exclude from the build when creating the manifest.

provDownloadBuild("3.0.22", {excludePaths:[`lib/fan/hvacExt.pod`]})

meta


def	func:provDownloadBuild
is	func
lib	lib:prov
name	provDownloadBuild
su	✓

supertypes


feature	Feature namespace of definitions formatted as `feature:name`
func	Axon function

def

func:provDownloadPatches

provDownloadPatches(build)

Download all the current patches for a SkySpark build from skyfoundry.com and create or update a provPatch manifest for it.

provDownloadPatches("3.0.22")

meta


def	func:provDownloadPatches
is	func
lib	lib:prov
name	provDownloadPatches
su	✓

supertypes


feature	Feature namespace of definitions formatted as `feature:name`
func	Axon function

def

func:provNewImage

provNewImage(arg, dis, tags: {})

Create a provImage from an ordered list of manifest recs. The image will have the given display name, and you can specify additional tags to apply to the image.

provNewImage([@build-3-0-22, @patch-3-0-22, @myOverlay], "Custom 3.0.22", {version:"3.0.22", foo})

meta


def	func:provNewImage
is	func
lib	lib:prov
name	provNewImage
su	✓

supertypes


feature	Feature namespace of definitions formatted as `feature:name`
func	Axon function

def

func:provSkySparkImage

provSkySparkImage(version, manifests: null)

Create a provImage for the given SkySpark build number. It will contain that build and the current patches. You may optionally pass an ordered list of additional manifests to include (append) to the image.

provSkySparkImage("3.0.22")

provSkySparkImage("3.0.22", [@myOverlay1, @myOverlay2])

meta


def	func:provSkySparkImage
is	func
lib	lib:prov
name	provSkySparkImage
su	✓

supertypes


feature	Feature namespace of definitions formatted as `feature:name`
func	Axon function

def

provBuild

The manifest contains all files for a SkySpark build. The manifest must also be tagged with a version.

meta


def	provBuild
is	provManifest
lib	lib:prov

usage


provBuild provManifest

supertypes


marker	Marker labels a dict with typing information
entity	Top-level dicts with a unique identifier
provManifest	A provisioning manifest

tags


dis	Display name for an entity
id	Defines the unique identifier of an entity in system using a `ref` value type
version	Version string formatted as decimal integers separated by a dot

def

provImage

The manifest is an image and can be used to deploy files to nodes in the cluster. The contents of an image manifest is a Grid where each row is an immutable snapshot of another provManifest type. Files will be deployed in the order specified in the grid.

meta


def	provImage
is	provManifest
lib	lib:prov

usage


provImage provManifest

supertypes


marker	Marker labels a dict with typing information
entity	Top-level dicts with a unique identifier
provManifest	A provisioning manifest

tags


dis	Display name for an entity
id	Defines the unique identifier of an entity in system using a `ref` value type
version	Version string formatted as decimal integers separated by a dot

def

provManifest

A provisioning manifest

meta


def	provManifest
is	entity
lib	lib:prov
mandatory	✓

supertypes


marker	Marker labels a dict with typing information
entity	Top-level dicts with a unique identifier

subtypes


provBuild	The manifest contains all files for a SkySpark build
provImage	The manifest is an image and can be used to deploy files to nodes in the cluster
provNode	The manifest contains files which should be applied to a specific node in the cluster
provOverlay	The manifest contains files which should be applied after the SkySpark build and patches
provPatch	The manifest contains patches for a SkySpark build
provPlatform	The manifest contains files which should be applied to a specific platform as determined by the `hostModel` tag

tags


dis	Display name for an entity
id	Defines the unique identifier of an entity in system using a `ref` value type

def

provNode

The manifest contains files which should be applied to a specific node in the cluster. The node is specififed via the nodeRef tag.

meta


def	provNode
is	provManifest
lib	lib:prov

usage


provNode provManifest

supertypes


marker	Marker labels a dict with typing information
entity	Top-level dicts with a unique identifier
provManifest	A provisioning manifest

tags


dis	Display name for an entity
id	Defines the unique identifier of an entity in system using a `ref` value type

def

provOverlay

The manifest contains files which should be applied after the SkySpark build and patches.

meta


def	provOverlay
is	provManifest
lib	lib:prov

usage


provOverlay provManifest

supertypes


marker	Marker labels a dict with typing information
entity	Top-level dicts with a unique identifier
provManifest	A provisioning manifest

tags


dis	Display name for an entity
id	Defines the unique identifier of an entity in system using a `ref` value type
version	Version string formatted as decimal integers separated by a dot

def

provPatch

The manifest contains patches for a SkySpark build. The manifest must also be tagged with a version.

meta


def	provPatch
is	provManifest
lib	lib:prov

usage


provPatch provManifest

supertypes


marker	Marker labels a dict with typing information
entity	Top-level dicts with a unique identifier
provManifest	A provisioning manifest

tags


dis	Display name for an entity
id	Defines the unique identifier of an entity in system using a `ref` value type
version	Version string formatted as decimal integers separated by a dot

def

provPlatform

The manifest contains files which should be applied to a specific platform as determined by the hostModel tag.

meta


def	provPlatform
is	provManifest
lib	lib:prov

usage


provPlatform provManifest

supertypes


marker	Marker labels a dict with typing information
entity	Top-level dicts with a unique identifier
provManifest	A provisioning manifest

tags


dis	Display name for an entity
id	Defines the unique identifier of an entity in system using a `ref` value type

def

view:prov

Cluster provisioning tool

meta


appName	prov
def	view:prov
disKey	ui::prov
icon	view
is	view
lib	lib:prov
order	100
su	✓

supertypes


feature	Feature namespace of definitions formatted as `feature:name`
view	Fresco UI view

src

view:   {inherit: "table"}
patch:  {action:"provUpdate"}
deploy: {action:"provExt::DeployAction.openDeploy" multi select}
cancel: {action:"provCancel" multi select}
restart: {action:"provRestart" multi select disKey:"ui::restart"}
data:   {expr: "provView()"}

def

view:provManifests

Manage provisioning manifests

meta


appName	prov
def	view:provManifests
disKey	ui::manifests
icon	view
is	view
lib	lib:prov
order	101
su	✓

supertypes


feature	Feature namespace of definitions formatted as `feature:name`
view	Fresco UI view

src

view:     {inherit: "table"}
contents: {action:"provManifestRead" disKey:"ui::contents" select}
trash:    {action:"recTrash"}
download: {action:"provExt::DownloadAction.download" select}
data:     {expr: "provManifestsView()"}

lib-prov 库参考 ​

目录（导读） ​

概述（中译） ​

API 正文（英文） ​

docs ​

apps ​

funcs ​

views ​

app:prov ​

lib prov ​

Overview ​

Setup ​

FileRepo ​

Prov Manifest ​

provBuild ​

provPatch ​

provOverlay ​

provNode ​

provImage ​

Creating Prov Manifests ​

Provc ​

Upload Action ​

Stub Nodes ​

Deployment ​

Patch Deployments ​

func:provDownloadBuild ​

provDownloadBuild(ver, opts: {}) ​

func:provDownloadPatches ​

provDownloadPatches(build) ​

func:provNewImage ​

provNewImage(arg, dis, tags: {}) ​

func:provSkySparkImage ​

provSkySparkImage(version, manifests: null) ​

provBuild ​

provImage ​

provManifest ​

provNode ​

provOverlay ​

provPatch ​

provPlatform ​

view:prov ​

src ​

view:provManifests ​

src ​

`lib-prov` 库参考

目录（导读）

概述（中译）

API 正文（英文）

docs

apps

funcs

views

app:prov

lib prov

Overview

Setup

FileRepo

Prov Manifest

provBuild

provPatch

provOverlay

provNode

provImage

Creating Prov Manifests

Provc

Upload Action

Stub Nodes

Deployment

Patch Deployments

func:provDownloadBuild

provDownloadBuild(ver, opts: {})

func:provDownloadPatches

provDownloadPatches(build)

func:provNewImage

provNewImage(arg, dis, tags: {})

func:provSkySparkImage

provSkySparkImage(version, manifests: null)

provBuild

provImage

provManifest

provNode

provOverlay

provPatch

provPlatform

view:prov

src

view:provManifests

src