FIN 5.3.1 中文文档

外观

hxPy ​

原文: https://finproducts.atlassian.net/wiki/spaces/JIDD3/pages/34793652293/FIN+5.3.1+hxPy 中文整理基于 FIN 5.3.1

导读： sys::IOErr: java.net.BindException: 权限被拒绝

以下为 API 参考，保留英文以便对照源码；目录已列出（英文标题便于检索）；概述已尽量中译。

目录（导读） ​

  • mixins
  • classes
  • docs
  • Overview
  • Prerequisites
  • Axon
  • Persistent Sessions
  • Data Marshalling
    • Grid Marshalling
  • Python Environment
    • IO
  • Docker in Docker
  • Slot Details
    • py
    • pyDefine
    • pyEval
    • pyExec
    • pyInit
    • pyTimeout
    • cur
    • make
    • close
    • define
    • eval
    • exec
    • init
    • kill
    • timeout

概述（中译） ​

这个 Python 库用于在沙箱化的 Docker 容器中执行 Python 代码。运行在此容器中的默认 Docker 镜像被配置为提供对多个 Python 库的访问，以便于科学计算和机器学习。

API 正文（英文） ​

Prerequisites ​

The Python library requires access to a Docker instance through use of the docker extension. Docker must have the hxpy image installed in order to execute Python code. The hxpy image can be pulled from the Haxall GitHub package repository:

> docker pull ghcr.io/haxall/hxpy:latest

If you are running SkySpark on a linux-based OS, then you must run SkySpark with Java 17+ in order to connect to the default docker daemon (this is because the default daemon must be accessed using a Unix Domain Socket, and Java didn't add support for that until Java 16). Note also that this Unix domain socket is typically owned by root:docker and you may need to add your user to the docker group in order to connect to the socket. If your user does not have access to the domain socket you will likely see an error like this:

sys::IOErr: java.net.BindException: Permission denied

Axon ​

To run Python code, you must first open a session. Opening a session causes a new Docker container to be created and started. This container is running the hxpy Python module which is responsible for handling the various commands you can issues on a session. Use the py() Axon function to create a session.

The general pattern for using a python session is:

  1. Create a session using py()
  2. Configure the session using funcs like pyDefine(), pyExec(), and pyTimeout()
  3. Evaluate an expression using pyEval() to get a result and close the session.

The following is a simple example demonstrating this pattern. In this example we create a session, define some variables, and evaluate an expression. After this code executes, the val variable will have the value 200.

val: py()
  .pyTimeout(10sec)
  .pyDefine("a", 100)
  .pyDefine("b", 2)
  .pyExec("def mult(x, y): return x * y\n")
  .pyEval("mult(a, b)")

Persistent Sessions ​

By default, a Python session is closed after a call to pyEval(). However, if you create a Python session in a task, then the session will remain opened until the task is killed/removed. There is no need to call taskLocalSet() to save the Python session returned by py(). The first call to py() in your task will create a new Python session. Subsequent message processed by your task will re-use the same session. You will typically want to use pyInit() to do any one-time configuration of your python session in your task:

// Example task
(msg) => do
  v: msg["v"]
  // This py() session will be remembered between task invocations
  r: py()
      .pyInit((session) => do
         // this code will only be invoked once for the session
         session.pyExec("def sq(x): return x * x\n")
       end)
      .pyDefine("v", v)
      .pyEval("sq(v)")
  echo("Got: " + r)
end

Data Marshalling ​

You can use the pyDefine() function to pass variables to the remote the Python instance for use in your code. The Python library will convert Haystack types to Python types as defined in the table below. Only the marked data types can be returned using pyEval().

Haystack    Python        pyEval()   Notes
--------    ------        --------   -----
null        None          ✓
Marker      Marker        ✓          hxpy.haystack.marker.Marker
NA          NA            ✓          hxpy.haystack.na.NA
Remove      Remove        ✓          hxpy.haystack.remove.Remove
Bool        bool          ✓
Number      int/float     ✓          Units not marshalled
Ref         Ref           ✓          hxpy.haystack.ref.Ref
Str         str           ✓
Uri         str
Time        datetime.time ✓
DateTime    datetime.date ✓
Buf         bytes         ✓
Dict        dict          ✓
List        list          ✓
Grid        Grid          ✓          hxpy.haystack.grid.Grid (see below)
MatrixGrid  numpy.ndarray ✓          see below

Grid Marshalling ​

The Python library has built-in support for marshalling Grids to Python types that are optimized and suitable for machine learning.

By default, a Grid is marshalled to an hxpy.haystack.grid.Grid type in Python. It is very common in data science applications to work with tabular data using the pandas.DataFrame type. You can get a DataFrame from a Grid in your Python code by using grid.to_dataframe(). You can return a pandas.DataFrame as the result of an pyEval() expression and it will automatically be converted to a Grid. You can also convert a DataFrame to a Grid using the Grid.from_dataframe(frame) static method.

py().pyDefine("grid", grid).pyEval("str(grid.to_dataframe())")

You can also convert a DataFrame to a Grid or GridBuilder using Grid.from_dataframe(frame) or GridBuilder.from_dataframe(frame) respectively. Use the latter when you need to add grid or column meta before creating the Grid.

Any Grid that is converted to a matrix using toMatrix() will be converted to a numpy.ndarray for use in your Python code.

py().pyDefine("data", toMatrix(g)).pyEval("type(data)")

Python Environment ​

The Docker container running Python will launch the hxpy module. Python code executed in this container has access to several common Python libraries that are useful for scientific computing and machine learning. The full list of available Python libraries is available here

IO ​

When the hxpy container is started, the current runtime project's io/ directory will be mounted into the container at /io/. This allows you to create/read files between the Haxall environment and the Python container.

// Write a simple text file to the mounted io/ directory
py()
  .pyExec("""def ioTest():
               with open("/io/example.txt", "w") as f:
                 f.write("Testing IO Example")
             """)
  .pyEval("ioTest()")

// read in Axon the contents of the file we just created in Python
s: ioReadStr(`io/example.txt`)

Docker in Docker ​

In some deployments, the Haxall runtime itself is running in a Docker container. To use the Python lib in this case it becomes necessary to run Docker within Docker. The Haxall Docker instance needs to create a Python Docker instance and connect to it.

You will need to create a Docker network for for your containers to run in. The Haxall runtime must be run in this network and you must use the network option of the py() func to have the python Docker instance also run in that network.

PyFuncs ​

const class PyFuncs : Obj

Axon functions for py

Methods

pyCreate a new PySession instance.
pyDefineDefine a variable to be available to python code running in the session.
pyEvalEvalue the given python statement in the session, and return the result.
pyExecExecute the given python code in the session and return the python session.
pyInitInitialize the python session by calling the given func to do any one-time setup of the python session.
pyTimeoutSet the timeout for pyEval().

Slot Details ​

py ​

@Axon { admin=true } static PySession py(Dict? opts := null)

Create a new PySession instance. Options:

  • image: name of the Docker image to run. By default, the lib will try to run the following images in this order:
    1. ghcr.io/haxall/hxpy:<ver> (where ver = current library Haxall version)
    2. ghcr.io/haxall/hxpy:latest
    3. ghcr.io/haxall/hxpy:main
  • network: the name of the Docker network you want the container to join.
  • logLevel: log level of the hxpy python process in Docker. Valid values are WARN, INFO, DEBUG, (default='WARN')

The default timeout for pyEval() is 5min. Use pyTimeout() to change this timeout.

Sessions created in the context of a task are persistent, meaning they will not be closed until the task is killed.

pyDefine ​

@Axon { admin=true } static PySession pyDefine(PySession py, Str name, Obj? val)

Define a variable to be available to python code running in the session.

pyEval ​

@Axon { admin=true } static Obj? pyEval(PySession py, Str stmt)

Evalue the given python statement in the session, and return the result. The session will be closed unless it is running in a task.

pyExec ​

@Axon { admin=true } static PySession pyExec(PySession py, Str code)

Execute the given python code in the session and return the python session. Note: python exec() does not return a value, so use pyEval() if you need the result of running a python statement. This method is primarily useful for declaring functions that you want available when using pyEval().

pyInit ​

@Axon { admin=true } static PySession pyInit(PySession py, Fn fn)

Initialize the python session by calling the given func to do any one-time setup of the python session. If pyInit() has already been called on this session, then the callback is not invoked.

Typically, this func is used in the context of a task since the python session in a task is persistent. This allows to do any one-time pyExec() or pyDefine() when the task is first creatd.

pyTimeout ​

@Axon { admin=true } static PySession pyTimeout(PySession py, Number? val)

Set the timeout for pyEval().

PyLib ​

const class PyLib : HxLib

Python library.

Constructors

makeConstruction

Methods

curConvenience to get the PyLib instance from the current context.

Slot Details ​

cur ​

static PyLib? cur(Bool checked := true)

Convenience to get the PyLib instance from the current context.

make ​

new make()

Construction

PySession ​

mixin PySession

Mixin for types that implement a python session.

Methods

closeClose the session, and return this
defineDefine a variable in local scope, and return this
evalEvaluate the expression and return the result.
execExecute the given code block, and return this
initIf the session has not been initialized yet, invoke the callback with this session to allow it to do one-time setup.
killKill the session - it can no longer be used again
timeoutSet the timeout for evaluating python expressions and return this

Slot Details ​

close ​

virtual This close()

Close the session, and return this

define ​

abstract This define(Str name, Obj? val)

Define a variable in local scope, and return this

eval ​

abstract Obj? eval(Str expr)

Evaluate the expression and return the result.

exec ​

abstract This exec(Str code)

Execute the given code block, and return this

init ​

abstract This init(|PySession| fn)

If the session has not been initialized yet, invoke the callback with this session to allow it to do one-time setup.

kill ​

virtual This kill()

Kill the session - it can no longer be used again

timeout ​

abstract This timeout(Duration? dur)

Set the timeout for evaluating python expressions and return this