Send HTTP Requests Using Agent Configuration

What is Agent Configuration?

LOC provides agents for operating not only internal data sources, but also external ones like HTTP endpoints, databases, FTP servers and SMTP servers. These is what makes LOC truely useful for building data pipelines across your organization.

However, these external sources often require authentication information (user account name and password, etc.), which may bring some issues:

• Hard-coding or pass unencrypted confidential data into logic would be a huge security risk.
• It would be a nightmare trying to maintain/update confidential data across numerous data processes.

Since v0.7.0, LOC provides Agent Configuration for authorised users to create and maintain connection profiles for HTTP, database, file sorage and mail agents. Here we'll learn how to create a configuration for your HTTP agent.

In order to use an agent configuration, you'll need to go through three steps:

1. Create a new agent configuration.
2. Add the configuration into a logic with a reference name.
3. Have an agent setup its client using the reference name.

Logic and Data Process Walkthrough

In this example we'll create a generic logic http-echo-response and a data process. The latter will be consisted with two other reusable logic.

Logic	Name	Purpose
Generic logic #1	payload-json-parser (source)	Read and parse JSON payload
Generic logic #2	http-echo-response	Invoke HTTP API with parsed payload and pass down the parsed response body
Aggregator logic	result-aggregator (source)	Finalise task result

http-echo-response (generic logic #2)

import {
    LoggingAgent,
    SessionStorageAgent,
    HttpAgent,
} from "@fstnetwork/loc-logic-sdk";

export async function run(ctx) {
    // read parsed JSON from session
    const parsed = await SessionStorageAgent.get("parsed");

    // create a HTTP client using a agent configuration reference
    const httpClient = await HttpAgent.acquire("http-echo-api-config");

    // log status
    LoggingAgent.info("HTTP client required successfully");

    // invoke HTTP API
    const response = await httpClient?.fetch("/anything", {
        method: "POST",
        headers: {
            "Content-Type": "application/json; charset=UTF-8",
            "X-Custom-Id": "Studio", // custom header
        },
        body: JSON.stringify(parsed),
    });

    // log status
    LoggingAgent.info("HTTP request sent");

    // if the response is HTTP 200 OK or 301 Move Permanently,
    // read and parse the response body
    let body = null;
    if (response.ok || response.status === 301) body = await response.json();

    // prepare result
    const result = {
        http_config_name: "http-echo-api-config",
        http_status: response.status,
        response_body: body,
    };

    // log result
    LoggingAgent.info(result);

    // store result in session
    await SessionStorageAgent.putJson("result", result);
}

export async function handleError(ctx, error) {
    // error logging
    LoggingAgent.error({
        error: true,
        errorMessage: error.message,
        stack: error.stack,
        taskId: ctx.task.taskId,
    });
}

Create a Configuration

For this tutorial, we'll use the HTTP agent to invoke https://httpbin.org/anything. This is a public API that simply returns anything you've sent to it, which is ideal for testing purposes.

In Studio, go to Administration then click the panel Agent Configuration. Right-click on the Unit to create a folder with a name:

Right-click on the folder and select Create Agent Configuration to create a new one:

In the configuration creation window, set the HTTP config as below:

1. Name of the configuration

2. Agent Configuration Type: HTTP Agent

3. Host is httpbin.org, scheme is HTTP, port is 80, prefix is empty.

4. You can add anythin in the description or to add additional custom headers.

tip

Custom headers added in configuration window and logic code will both be included in the HTTP request.

Finally click Create.

Add a Configuration to a Linked Logic

A configuration has to be manually added to a logic (which has to be linked in a data process) to be accessible. The HTTP Agent would be able to load the configuration and returns a working HTTP client object.

Now open the data process and modify it to create a revision:

Move your mouse cursor to the upper=right corner of http-echo-response logic, either click + or move the cursor to ⋮ then select + Add Configuration:

info

The Variable panel above is for setting up logic environment variables. See Logic Variable Agent for how to access them in logic.

Click Add a row on the right, then click on the Agent Configuration Name to find your configuration:

Give the configuration a referenece name (in this case we also named it http-echo-response) then click OK:

tip

HTTP Agent loads configuration from its reference name, not the configuration name.

You should see a configuration has been added to the logic (with a AC icon as well as Agent Configuration table on the right):

Test the HTTP Agent

Now you can try to invoke the data process with a JSON payload, either using single data process execution or via an API route.

For example, if you send this as your payload:

{
    "name": "Zaphod Beeblebrox",
    "title": "Ex-Galactic President",
    "number-of-heads": 2
}

The task result returned shoudl be like this, which contains the full HTTP response body from the HTTP endpoint:

{
    "status":"ok",
    "taskId":{
        "id":"4bXSumE7Dwl65kSYej7rng",
        "executionId":"Y-CRtV-0oJm0EHySDPysPw"
    },
    "response":{
        "http_config_name":"http-echo-api-config",
        "http_status":200,
        "response_body":{
            ...
            "headers":{
                "Accept":"*/*",
                "Accept-Encoding":"gzip, br",
                "Accept-Language":"*",
                "Content-Length":"80",
                "Content-Type":"application/json",
                "Host":"httpbin.org",
                ...
                "X-Custom-Id":"Studio",
                ...
            },
            "json":{
                "name":"Zaphod Beeblebrox",
                "number-of-heads":2,
                "title":"Ex-Galactic President"
            },
            "method":"POST",
            "origin":"35.81.150.116",
            "url":"http://httpbin.org/anything"
        }
    }
}

This shows that the HTTP agent configuration is setup correctly and the HTTP agent works without issue.