Advanced Usages of AI Oracle

In this tutorial, we’ll explore advanced techniques for interacting with the AI Oracle. Specifically, we’ll dive into topics like Nested Inference, Batch Inference, and Data Availability (DA) Options, giving you a deeper understanding of these features and how to leverage them effectively.

1. Nested Inference

A user can perform nested inference by initiating a second inference based on the result of the first inference within a smart contract. This action can be completed atomically and is not restricted to a two-step function.

Nested Inference Use Cases

Some of the use cases for a nested inference call include:

• generating a prompt with LLM for AIGC (AI Generated Content) NFT

• extracting data from a data set, then generate visual data with different models

• adding transcript to a video, then translate it to different languages with different models

For demo purposes we built a farcaster frame that uses ORA's AI Oracle.

Implementing Nested Inference

The idea of Nested Inference contract is to execute multiple inference requests in 1 transaction. We'll modify Prompt contract to support nested inference request. In our example, it will call Llama3 model first, then use inference result as the prompt to another request to StableDiffusion model.

The main goal of this tutorial is to understand what changes we need to make to Prompt contract in order to implement logic for various use cases.

Implementation Steps

1. modify CalculateAIResult method to support multiple requests

2. modify aiOracleCallback with the logic to handle second inference request

💡 When estimating gas cost for the callback, we should take both models into the consideration.

CalculateAIResult

As we now have additional function parameter for second model id. Not that we encode and forward model2Id as a callback data in aiOracle.requestCallback call.

 function calculateAIResult(uint256 model1Id, uint256 model2Id, string calldata model1Prompt) payable external returns (uint256) {
    bytes memory input = bytes(model1Prompt);
    uint256 model1Fee = estimateFee(model1Id);
    uint256 requestId = aiOracle.requestCallback{value: model1Fee}(
        model1Id, input, address(this), callbackGasLimit[model1Id], abi.encode(model2Id)
    );
    AIOracleRequest storage request = requests[requestId];
    request.input = input;
    request.sender = msg.sender;
    request.modelId = model1Id;
    emit promptRequest(requestId, msg.sender, model1Id, model1Prompt);
    return requestId;
}

aiOracleCallback

The main change here is the within "if" block. If the callback data (model2Id) is returned, we want to execute second inference request to the AI Oracle.

Output from the first inference call, will be passed to second one. This allows for interesting use cases, where you can combine text-to-text (eg. Llama3) and text-to-image (eg. Stable-Diffusion) models.

If nested inference call is not successful the whole function will revert.

💡 When interacting with the contract from the client side, we need to pass cumulative fee (for both models), then for each inference call we need to pass part of that cumulative fee. This is why we are calling estimateFee for model2Id.

function aiOracleCallback(uint256 requestId, bytes calldata output, bytes calldata callbackData) external payable override onlyAIOracleCallback() {
    AIOracleRequest storage request = requests[requestId];
    require(request.sender != address(0), "request does not exist");
    request.output = output;
    prompts[request.modelId][string(request.input)] = string(output);

    //if callbackData is not empty decode it and call another inference
    if(callbackData.length != 0){
        (uint256 model2Id) = abi.decode(callbackData, (uint256));
        uint256 model2Fee = estimateFee(model2Id);

        (bool success, bytes memory data) = address(aiOracle).call{value: model2Fee}(abi.encodeWithSignature("requestCallback(uint256,bytes,address,uint64,bytes)", model2Id, output, address(this), callbackGasLimit[model2Id], ""));
        require(success, "failed to call nested inference");

        (uint256 rid) = abi.decode(data, (uint256));
        AIOracleRequest storage recursiveRequest = requests[rid];
        recursiveRequest.input = output;
        recursiveRequest.sender = msg.sender;
        recursiveRequest.modelId = model2Id;
        emit promptRequest(rid, msg.sender, model2Id, "");
    }

    emit promptsUpdated(requestId, request.modelId, string(request.input), string(output), callbackData);
}

Interaction with Contract

This is an example of contract interaction from Foundry testing environment. Note that we're estimating fee for both models and passing cumulative amount during the function call (we're passing slightly more to ensure that the call will execute if the gas price changes).

PromptNestedInference prompt = new PromptNestedInference(IAIOracle(OAO_PROXY));

uint256 stableDiffusionFee = prompt.estimateFee(STABLE_DIFFUSION_ID);
uint256 llamaFee = prompt.estimateFee(LLAMA_ID);

uint256 requestId = prompt.calculateAIResult{value: ((stableDiffusionFee + llamaFee)*11/10)}(STABLE_DIFFUSION_ID, LLAMA_ID, SD_PROMPT);

Conclusion

You can also check the full implementation of PromptNestedInference.

2. Batch Inference

The Batch Inference feature enables sending multiple inference requests within a single transaction, reducing costs by saving on network fees and improving the user experience. This bulk processing allows for more efficient handling of requests and results, making it easier to manage the state of multiple queries simultaneously.

Some of the use cases might be:

• AIGC NFT marketplace - creating a whole AIGC NFT collection with just one transaction, instead of creating those with many transactions

• Apps that need to handle requests simultaneously - Good example would be a recommendation system or chatbot with high TPS

Pricing

Model fee required for batch inference request is calculated by multiplying single model fee with batch size (batchSize * model.fee). This fee covers the operational costs of running AI models, with a portion contributing to protocol revenue. For details on the required fees for each model, visit the References page.

Callback transaction fee is needed for AI Oracle to submit callback transaction. It's calculated by multiplying current gas price with callback gas limit for invoked model (gasPrice * callbackGasLimit).

Request transaction fee is regular blockchain fee needed to request inference by invoking aiOracle.requestCallback method.

Total fee is calculated as sum of Model fee, Callback transaction fee and Request transaction fee.

Implementing Batch Inference

In order to initiate batch inference request, we will interact with requestBatchInference method. This method takes additional batchSize parameter, which specifies the amount of requests to the AI Oracle.

function calculateAIResult(uint256 modelId, string calldata prompt, uint256 batchSize) payable external {
    bytes memory input = bytes(prompt);
    bytes memory callbackData = bytes("");
    address callbackAddress = address(this);
    uint256 requestId = aiOracle.requestBatchInference{value: msg.value}(
        batchSize, modelId, input, callbackAddress, callbackGasLimit[modelId], callbackData, IAIOracle.DA(0), IAIOracle.DA(0)
    );
}

Note that we'll need to pass more gas to cover AI Oracle callback execution, depending on the batchSize (check out Pricing). For this purpose we can implement estimateFeeBatch function in our Prompt contract. This method will interact with estimateFeeBatch method from AIOracle.sol.

//Prompt.sol
function estimateFeeBatch(uint256 modelId) public view returns (uint256) {
    return aiOracle.estimateFee(modelId, callbackGasLimit[modelId]);
}

//AIOracle.sol
function estimateFeeBatch(uint256 modelId, uint256 gasLimit, uint256 batchSize) public view ifModelExists(modelId) returns (uint256) {
    ModelData storage model = models[modelId];
    return batchSize * model.fee + gasPrice * gasLimit;
}

Prompt Format

Input for batch inference should be structured as a string representing an array of prompt and seed values.

Prompt - string value that is mandatory in order to prompt AI Oracle.

Seed – an optional numeric value that, when used, allows you to generate slightly varied responses for the same prompt.

[
    // Batch inference requests with required "prompt" and optional "seed" or "seed_range".
    {
        "prompt": "prompt1",  // Mandatory prompt
        "seed": 1             // Optional seed for response variation
    },
    {
        "prompt": "prompt2",
        "seed": 2
    },
    {
        "prompt": "prompt3"    // No seed provided, default behavior
    },
    {
        "prompt": "prompt4",
        "seed_range": [1, 2]   // Seed range for controlled variation
    }
]

Output Format

Result of Batch inference call is a dot separated list of inference results.

result1.result2.result3...

Example

This is the prompt for interacting with Stable Diffusion model:

[{"prompt":"generate image of an elephant","seed":1},{"prompt":"generate image of an elephant","seed":2}]

Result is dot separated list of ipfs CIDs:

.Qma3gR6z9dtSQ3fsoSUwQtHnNMYZEHhuJx3DBqx677HPaj.QmemPcnzdyigx1XnAvskVKmEhnsivBuE3Qkp3jJqfdWWxC

Interaction with Batch Inference

When performing batch inference with the AI Oracle, ensure the prompt is following the standard format. Below is a simple script for interacting with batch inference:

import { Web3 } from "web3";

const web3 = new Web3(process.env.RPC_URL);

const wallet = web3.eth.accounts.wallet.add(process.env.PRIVATE_KEY); // Make sure you have funds

const contract = new web3.eth.Contract(batchInference_abi, batchInference_address);

const prompt = `[{"prompt":"what's the best fruit", "seed":1},{"prompt":"what's the best fruit", "seed":2},{"prompt":"kiwi"},{"prompt":"dog", "seed_range":[1,2]}]`

const fee = Number(await contract.methods.estimateFeeBatch(11, 5).call());

const totalFee = (fee*11/10)

const tx = await contract.methods.requestBatchInference(11, prompt, 5).send({from: wallet[0].address, value: totalFee});
console.log("Tx: ", tx)

setTimeout(async () => {
    const result = await contract.methods.prompts(11, prompt).call();
    console.log("Result: ", result)
}, 30000);

To test it, you need to:

1. create new javascript file

2. copy the script and add env variables

3. create and deploy prompt contract that supports batch inference

4. add values to batchInference_abi and batchInference_address

Prompt examples

// Example 1: Batch with identical prompts but different seeds for varied responses
const prompt1 = `[{"prompt":"hello","seed":1},{"prompt":"hello","seed":2}]`

// Example 2: Batch with different prompts, no seeds specified, uses default behavior
const prompt2 = `[{"prompt":"hello"},{"prompt":"hello world"}]`

// Example 3: Batch with same question but varied seeds, and different prompts with a seed range
const prompt3 = `[{"prompt":"what's the best fruit", "seed":1},{"prompt":"what's the best fruit", "seed":2},{"prompt":"kiwi"},{"prompt":"dog","seed_range":[1,2]}]`

// Example 4: Batch with identical prompt but different seeds, plus a seed range for a specific prompt
const prompt4 = `[{"prompt":"apple", "seed":1}, {"prompt":"apple", "seed":2}, {"prompt":"banana"}, {"prompt":"cat", "seed_range": [1,100] }]`

Conclusion

That's it! With few simple changes to the Prompt contract we utilised batch inference feature. This allowed us to get multiple responses with only one transaction.