lib-clone 库参考

原文: https://finproducts.atlassian.net/wiki/spaces/JIDD3/pages/34793521313/FIN+5.3.1+lib-clone 中文整理基于 FIN 5.3.1

导读： 克隆基于记录的复制扩展

以下为 API 参考，保留英文以便对照源码；目录已列出（英文标题便于检索）；概述已尽量中译。

---

目录（导读）

• lib
• meta
• docs
• tags
• apps
• funcs
• views
• def
• supertypes
• tags
• enum
• Overview
• Terminology
• Clone Sets
• Clones
• Clone Refs
• Clone Orphans
• Protocol
• Security
• Tuning
• Overlaps
• Over Max
• Sync History
• cloneReadOrphans()
• cloneRemoveOrphans(recs)
• cloneResubscribe(cloneSets)
• cloneSyncHis(points, span: null)
  • src

---

概述（中译）

克隆基于记录的复制扩展

---

API 正文（英文）

docs

Overview • Terminology • Clone Sets • Clones • Clone Refs • Clone Orphans • Protocol • Security • Tuning • Overlaps • Over Max • Sync History

---

tags

clone	Marks a record as a clone
cloneFilter	Filter string for clone set data
cloneMod	Modified timestamp of the clone's progenitor
cloneProj	Project name for clone set data
cloneSet	Defines a clone set to subscribe to records in a remote project
cloneSetRef	Clone set which manages the clone
cloneSetStatus	Current status of a cloneSet

---

apps

clone	Manage clone replication

---

funcs

cloneReadOrphans	Read orphaned clones from the database as a grid
cloneRemoveOrphans	Remove clone recs which no longer have a reference to a valid cloneSet
cloneResubscribe	Force a re-subscription of one or more client side clone sets
cloneSyncHis	Perform remote sync of history data for given clone points

---

views

cloneDebugClient	Debug client side cloning
cloneDebugServer	Debug server side cloning
cloneOrphans	View and delete cloning orphans
cloneSets	Manage clone sets
cloneSettings	Settings for the clone extension

---

---

def

app:clone

Manage clone replication

---

meta

def	app:clone
disKey	ui::clone
icon	cloneDb
is	app
lib	lib:clone

---

supertypes

feature	Feature namespace of definitions formatted as feature:name
app	UI application grouping of views

---

---

def

clone

Marks a record as a clone. See docs for further details.

---

meta

def	clone
is	marker
lib	lib:clone

---

supertypes

marker	Marker labels a dict with typing information

---

tags

cloneMod	Modified timestamp of the clone's progenitor
cloneSetRef	Clone set which manages the clone

---

---

def

cloneFilter

Filter string for clone set data

---

meta

def	cloneFilter
is	filterStr
lib	lib:clone
tagOn	cloneSet

---

supertypes

val	Data value type
scalar	Scalar is an atomic value kind
str	Unicode string of characters
filterStr	Haystack filter string

---

---

def

cloneMod

Modified timestamp of the clone's progenitor

---

meta

def	cloneMod
is	dateTime
lib	lib:clone
tagOn	clone

---

supertypes

val	Data value type
scalar	Scalar is an atomic value kind
dateTime	ISO 8601 timestamp followed by timezone identifier

---

---

def

cloneProj

Project name for clone set data. For Haxall nodes this is the node identifier (without the n: prefix).

---

meta

def	cloneProj
is	str
lib	lib:clone
tagOn	cloneSet

---

supertypes

val	Data value type
scalar	Scalar is an atomic value kind
str	Unicode string of characters

---

---

def

cloneSet

Defines a clone set to subscribe to records in a remote project. See docs for further details.

---

meta

def	cloneSet
is	entity
lib	lib:clone
mandatory	✓

---

supertypes

marker	Marker labels a dict with typing information
entity	Top-level dicts with a unique identifier

---

tags

cloneFilter	Filter string for clone set data
cloneProj	Project name for clone set data
cloneSetStatus	Current status of a cloneSet
dis	Display name for an entity
id	Defines the unique identifier of an entity in system using a ref value type

---

---

def

cloneSetRef

Clone set which manages the clone

---

meta

def	cloneSetRef
is	ref
lib	lib:clone
of	cloneSet
tagOn	clone

---

supertypes

val	Data value type
scalar	Scalar is an atomic value kind
ref	Reference to an entity

---

---

def

cloneSetStatus

Current status of a cloneSet

---

meta

def	cloneSetStatus
enum	See below
is	str
lib	lib:clone
tagOn	cloneSet
transient	✓

---

enum

ok	Clone set is okay and active
unresolved	Cannot resolve project name to cluster node
down	Communication or network problem
warn	Configuration problem causing partial failure
fault	Configuration problem
disabled	Manual disable

---

supertypes

val	Data value type
scalar	Scalar is an atomic value kind
str	Unicode string of characters

---

lib clone

Overview

Cloning is a cluster feature which enables fine-grained record based replication. Cloning is significantly different from database replication that is used to replicate the entire Folio database including all the history data, spark cache, and arc documents. Instead cloning is used to replicate specific records from one project to another project.

Salient features of the cloning extension include:

• Synchronized replication of records between project
• Only persistent tags are synchronized
• Records to replicate are identified using filters
• Leverages Arcbeam clustering
• Automatic synchronization of changes (typically within a few seconds)
• Supports synchronization from Haxall or SkySpark nodes
• Client support only in SkySpark
• Clones can be used as targets for rule engine
• Synchronize history data via cloneSyncHis function

Terminology

The following terminology is used with the cloning framework:

• server project: the project with the source records to replicate from
• client project: project we wish to replicate to
• clone set: record in client project used to configure the server project and filter of records to replicate
• progenitor: a single record in the server project replicated to one or more client projects
• clone: replicated copy record in the client project

Clone Sets

Clone sets are records configured in the client project to subscribe to records in remote projects. Clone sets are configured with the following tags:

• cloneSet: marker tag
• cloneProj: project name or node id for remote project
• cloneFilter: filter used to subscribe to matching records

If you are cloning from a Haxall node, then the cloneProj tag should be set to the clustering node id without the n: prefix.

Clones

Clones are records in the client project which are automatically kept in-sync with their respective progenitors. Most tags in a clone are mirrored from their progenitor with the following exceptions:

• clone: marker tag is added to all clones
• cloneSetRef: reference to parent clone set which defines source project
• cloneMod: used to track of their progenitor's mod timestamp for sync status
• all references are relativized to the local project (discussed below)

Attempts to modify a clone record will raise an exception. This means they cannot be edited in the UI, nor may be they be changed via the commit() function. The only way to change a clone is it to update its progenitor which will in turn be reflected in the clone through the synchronization mechanism. Note if the clone extension is not enabled, then clones may be modified as normal records.

Clone Refs

The ref tags of progenitors are relativized to the local project. This includes the id tag as well as relationship tags such as siteRef, equipRef, etc:

Server     Progenitor            Clone
--------   -------------------   ------------------
SkySpark   @p:server:r:abc-123   @p:client:r:abc-123
Haxall     @abc-123              @p:client:r:abc-123

Clone Orphans

If you remove a cloneSet rec from the database, then its clones are left in an orphaned state. Because clones cannot be modified through standard tools such as commit(), you must use specialized functions to clean up orphaned clones. Use the cloneReadOrphans() to read all the orphaned records. Use cloneRemoveOrphans() to remove orphans from the database:

// read all orphans and then remove them from database
cloneReadOrphans().cloneRemoveOrphans

Protocol

The cloning extension utilizes a simple protocol over Arcbeam to efficiently maintain synchronization between projects. Understanding how the protocol works under the coves is necessary for tuning and diagnostics. There are three different exchanges between the client and server projects:

1. Sub: subscription of a client's clone sets with the server project
2. Sync: compare manifest of a client's clone set with the server project
3. Updates: events sent by the server to the client with updated records

The sub or subscription request is sent periodically by the client to the server project to register the client's clone sets. The server uses the subscription to maintain a registry of client clone sets for where it needs to push updates. During the subscription phase the client and server both compute a SHA-1 digest of their local records. If a client detects that a given clone set is out of date, then a sync request is initiated.

The sync request is sent by a client to the server for each clone set it detects is out of date from a subscription request. The request includes a manifest of every id and mod timestamp it has locally. The server uses this information to queue up any out of date records to push to the client.

The server pushes modified records in the clone set to the client via update events. During initialization, updates are pushed from the sync request. But once the initial synchronization is complete, the server will automatically push updates whenever it detects local changes. Servers maintain an obsCommits subscription on the clone set filter to listen for changes to the set.

Security

If you do not wish a given project to be a clone server, then do not enable the cloning extension. Projects without the clone extension enabled cannot be used as a server and clients will receive a subscription errror.

Arcbeam nodes which are set to isolation mode can be used as a clone server. However if a node is isolated then it cannot be used as a clone client.

Tuning

In a typical steady state case, the only exchange should be periodic subscription requests. If no changes have been made then the digests should match and this mechanism efficiently ensures client projects are up-to-date. Subscription requests are throttled via the subMinTime and subMaxTime settings.

By default a subscription is issued whenever a node transitions from down to up as long the last subscription was older than subMinTime. This ensures that we immediately check for changes which might have been made while the connection was down. However, this might be an expensive check for unstable Arcbeam connections which are going up and down; in that case consider turning off this feature with the subOnUp flag in the settings.

The updateFreq setting is used to throttle how fast server side changes are pushed to the client via update events. For records which don't change frequently then using a few seconds ensures rapid synchronization. However, if you have records changing often then consider tuning this frequency higher to save bandwidth and processing.

In terms of performance the goal is to avoid the sync request - it is very expensive for large clone sets. Ideal operation is to perform an initial synchronization, then only perform periodic subscriptions to ensure everything is up-to-date. In the steady state case with a stable Arcbeam connection then re-syncs should be rare.

Overlaps

It is a configuration error when two or more different clone sets to the same project match the same records. We call this condition overlaps. Because a given clone rec can only be managed by one clone set, this condition puts the client project into an indeterminate state. When this condition is flagged as a warning, it is imperative to fix your configuration to ensure each progenitor is matched by only one client clone set filter. This condition will cause your system to thrash with repeated synchronization requests that will never resolve. Note this same condition will occur if cloning recs from different projects which share the same id.

Over Max

Clone set size has a large impact on performance. Therefore we limit the size of a single clone set to 2500 records. If the system exceeds this limit then you will see the overMax flag in the debug views. Once you go over the limit, the client project's state becomes indeterminate. The first 2500 records sorted by id will continue to be synchronized. But in most cases the client project will end up with orphaned clones that will not be synchronized. The only safe way to recover from this issue is to delete your cloneSet and its orphaned records, then recreate the cloneSet from scratch using a more narrow filter.

Sync History

You can synchronize history data from cloned points into SkySpark using the cloneSyncHis() function. This function works just like connSyncHis(), except it leverages your cloning Arcbeam channel instead of a connector. For example you can setup a task to periodically sync all your cloned points using an expression like this:

readAll(clone and point and his).cloneSyncHis(null)

---

---

def

func:cloneReadOrphans

cloneReadOrphans()

Read orphaned clones from the database as a grid. A clone is considered orphaned if its cloneSetRef references a rec not found or in the trash.

---

meta

admin	✓
def	func:cloneReadOrphans
is	func
lib	lib:clone
name	cloneReadOrphans

---

supertypes

feature	Feature namespace of definitions formatted as feature:name
func	Axon function

---

---

def

func:cloneRemoveOrphans

cloneRemoveOrphans(recs)

Remove clone recs which no longer have a reference to a valid cloneSet. A clone is considered orphaned if its cloneSetRef references a rec not found or in the trash. See cloneReadOrphans() to read all the orphans from the database. If any recs specified are not orphans then raise an exception. If this function cannot be called in the case of a licensing error, then you can call cloneRemoveOrphansFailSafe() instead.

---

meta

admin	✓
def	func:cloneRemoveOrphans
is	func
lib	lib:clone
name	cloneRemoveOrphans

---

supertypes

feature	Feature namespace of definitions formatted as feature:name
func	Axon function

---

---

def

func:cloneResubscribe

cloneResubscribe(cloneSets)

Force a re-subscription of one or more client side clone sets. The minimum subscription time from settings is ignored and a new subscription is issued as soon as possible.

---

meta

def	func:cloneResubscribe
is	func
lib	lib:clone
name	cloneResubscribe

---

supertypes

feature	Feature namespace of definitions formatted as feature:name
func	Axon function

---

---

def

func:cloneSyncHis

cloneSyncHis(points, span: null)

Perform remote sync of history data for given clone points. The points parameter may be anything acceptable by toRecIdList(), but each point must be a clone. The span parameter is anything acceptable by toSpan(). Or pass null for span to perform a sync for items after the point's hisEnd. This func blocks the calling thread until each point is synchronized one by one. Normally it should only be called within a task. The result from this function is undefined. The history sync is based on the raw history data from the remote system - as if calling hisRead with the {raw} option.

Examples:

readAll(clone and point and his).cloneSyncHis(null)

---

meta

admin	✓
def	func:cloneSyncHis
is	func
lib	lib:clone
name	cloneSyncHis

---

supertypes

feature	Feature namespace of definitions formatted as feature:name
func	Axon function

---

---

def

view:cloneDebugClient

Debug client side cloning

---

meta

admin	✓
appName	clone
def	view:cloneDebugClient
disMacro	$ui::debug $ui::client
icon	view
is	view
lib	lib:clone

---

supertypes

feature	Feature namespace of definitions formatted as feature:name
view	Fresco UI view

---

src

view:        { inherit:"table" }
routes:      { action:"cloneClientDebugRoutes" }
details:     { action:"cloneClientDetails" select }
recs:        { action:"cloneClientDebugRecs" select }
resubscribe: { action:"cloneResubscribe" select multi }
data:        { expr:"cloneClientDebug()" }

---

---

def

view:cloneDebugServer

Debug server side cloning

---

meta

admin	✓
appName	clone
def	view:cloneDebugServer
disMacro	$ui::debug $ui::server
icon	view
is	view
lib	lib:clone

---

supertypes

feature	Feature namespace of definitions formatted as feature:name
view	Fresco UI view

---

src

view:    { inherit:"table" }
details: { action:"cloneServerDetails" select }
recs:    { action:"cloneServerDebugRecs" select }
data:    { expr:"cloneServerDebug()" }

---

---

def

view:cloneOrphans

View and delete cloning orphans

---

meta

admin	✓
appName	clone
def	view:cloneOrphans
disMacro	$ui::orphans
icon	view
is	view
lib	lib:clone

---

supertypes

feature	Feature namespace of definitions formatted as feature:name
view	Fresco UI view

---

src

view:   { inherit:"table" }
delete: { action:"cloneRemoveOrphans" select multi }
data:   { expr:"cloneReadOrphans()" }

---

---

def

view:cloneSets

Manage clone sets

---

meta

admin	✓
appName	clone
def	view:cloneSets
defName	cloneSet
disKey	ui::sets
icon	view
is	view
lib	lib:clone
order	100

---

supertypes

feature	Feature namespace of definitions formatted as feature:name
view	Fresco UI view

---

src

view:    { inherit:"table" }
search:  { input var kind:"Str" defVal:"" }
new:     { action:"recNew" }
edit:    { action:"recEdit" }
delete:  { action:"recDelete"}
enable:  { action:"recEnable" }
disable: { action:"recDisable" }
data:    { expr:"readAll(cloneSet, { sort, search:{{search}} })" }

---

---

def

view:cloneSettings

Settings for the clone extension

---

meta

admin	✓
appName	settings
def	view:cloneSettings
disKey	ui::clone
icon	view
is	view
lib	lib:clone

---

supertypes

feature	Feature namespace of definitions formatted as feature:name
view	Fresco UI view

---

src

view:   {inherit:"form"}
submit: {action:"extSettingsCommit"}
data:   {expr:"extSettings(\"clone\")"}