Skip to main content

Core components

Core Components​

1. ULLMAIClientComponent​

The main component for AI communication and session management.

Connection Management​

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI")
void ConnectToAI(const FString& Provider,
                    const FString& Model, 
                    const FString& Instructions, 
                    const TArray<FString>& Modalities,
                    float Temperature = 0.8f,
                    const FString& Voice = "");
// Provider: "OpenAI", "LocalAI", or "Grok"
// Modalities: ["text"], ["audio"], or ["text", "audio"]
// Voice: REQUIRED for Grok (locks after first session update)
//        Optional for OpenAI/LocalAI (can change via StartVoiceMode)

For Grok provider, the Voice parameter is REQUIRED and cannot be changed without reconnecting. Grok locks the voice after the first session update. If not specified, defaults to "Ara".

OpenAI and LocalAI do not have this limitation - voice can be changed mid-session via StartVoiceMode() .

Session State Functions​

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI", BlueprintPure)
bool IsAISession() const;
// True if connected to any AI provider

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI", BlueprintPure)
bool IsAISessionReady() const;
// True if session is ready for communication

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket", BlueprintPure)
ELLMAISessionState GetCurrentSessionState() const;

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket", BlueprintPure)
bool IsAIActivelyResponding() const;

Basic WebSocket Connection​

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket")
void Connect(const FString& URL, const FString& Protocol = TEXT(""));

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket")
void Disconnect();

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket", BlueprintPure)
bool IsConnected() const;

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket", BlueprintPure)
bool IsConnecting() const;

Provider-Specific (Internal)​

// These are called internally by ConnectToAI
void ConnectToOpenAI(const FString& Model, 
                         const FString& Instructions, 
                         const TArray<FString>& Modalities,
                         float Temperature = 0.8f,
                         const FString& Voice = "");

void ConnectToLocalAI(const FString& Model, 
                         const FString& Instructions, 
                         const TArray<FString>& Modalities,
                         float Temperature = 0.8f,
                         const FString& Voice = "");

void ConnectToGrok(const FString& Model, 
                         const FString& Instructions, 
                         const TArray<FString>& Modalities,
                         float Temperature = 0.8f,
                         const FString& Voice = "Ara");

See the Provider comparison table on the Providers page.

Session State Enum​

ELLMAISessionState​

UENUM(BlueprintType)
enum class ELLMAISessionState : uint8
{
    Disconnected,
    Connecting,
    Connected,
    SessionCreating,
    SessionReady,
    VoiceStarting,
    VoiceActive,
    Error
};

Voice Communication​

Voice Mode Control​

UFUNCTION(BlueprintCallable, Category = "LLMAI|Voice")
bool StartVoiceMode(const FString& Voice = TEXT(""),
                       const FString& TTSModel = TEXT("")); // v2.0: TTSModel param

UFUNCTION(BlueprintCallable, Category = "LLMAI|Voice")
bool EndVoiceMode(bool bClearConversation = true);

Audio Data Transmission​

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI")
void SendAudioToAI(const TArray<float>& AudioSamples);

UFUNCTION(BlueprintCallable, Category = "LLMAI|Voice")
void CommitAudioBuffer();

UFUNCTION(BlueprintCallable, Category = "LLMAI|Voice")
void TruncateAudioBuffer(int32 AudioEndMs = 0);

Microphone Control​

Manual Mute Control​

UFUNCTION(BlueprintCallable, Category = "LLMAI|Voice")
void SetMicrophoneMuted(bool bMuted);

UFUNCTION(BlueprintCallable, Category = "LLMAI|Voice")
void ToggleMicrophoneMute();

UFUNCTION(BlueprintPure, Category = "LLMAI|Voice")
bool IsMicrophoneMuted() const;

Auto-Gating Status​

UFUNCTION(BlueprintPure, Category = "LLMAI|Voice")
bool IsMicrophoneAutoGated() const;
// Returns true when mic is auto-muted because AI is speaking

UFUNCTION(BlueprintCallable, Category = "LLMAI|Voice")
void StopBufferedAudioPlayback();
// Stop playback when AI finished but audio still buffered

Text Communication​

Provider-Agnostic Text (Recommended)​

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI")
void SendTextToAI(const FString& Text, 
                      bool bUseTextModality = true, 
                      bool bUseAudioModality = true);

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI")
void AddContextToAI(const FString& ContextText);

AI Response Control​

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket")
void TriggerAIResponse(const FString& Instructions = TEXT(""));

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket")
void InterruptAIResponse();

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket")
void CancelResponse();

Modality-Specific Response​

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket")
void TriggerAIResponseWithModalities(
    const TArray<FString>& Modalities,
    const FString& Instructions = TEXT(""));

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket")
void TriggerTextOnlyResponse(const FString& Instructions = TEXT(""));

UFUNCTION(BlueprintCallable, Category = "LLMAI|Voice")
void TriggerAudioOnlyResponse(const FString& Instructions = TEXT(""));

Legacy Provider-Specific Functions​

// These still work but use generic versions above instead
void SendTextToOpenAI(const FString& Text, 
                           bool bUseTextModality = true, 
                           bool bUseAudioModality = true);

void AddContextToOpenAI(const FString& ContextText);
void SendTextToLocalAI(...);  // Same signature
void AddContextToLocalAI(...); // Same signature

Function Calling​

Function Registration​

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI Functions")
void RegisterAIFunction(const FLLMFunctionDefinition& FunctionDefinition);

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI Functions")
void UnregisterAIFunction(const FString& FunctionName);

Function Call Handling​

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI Functions")
void SendAIFunctionCallResult(const FString& CallId, 
                                    const FString& Result);

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI Functions", BlueprintPure)
TArray<FString> GetRegisteredAIFunctionNames() const;

Function Profiles​

Profile Management​

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI Function Profiles")
void RegisterFunctionProfile(ULLMAIFunctionProfile* Profile);

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI Function Profiles")
void RegisterFunctionDefinitionAsset(ULLMAIFunctionDefinitionAsset* FunctionAsset);

Profile Queries​

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI Function Profiles", BlueprintPure)
bool IsFunctionProfileLoaded(const FString& ProfileName) const;

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI Function Profiles", BlueprintPure)
TArray<FString> GetLoadedProfileNames() const;

Audio Setup​

Voice Capture & Playback​

UFUNCTION(BlueprintCallable, Category = "LLMAI|Voice")
void SetupVoiceCapture(UAudioCaptureComponent* MicComponent = nullptr);

UFUNCTION(BlueprintCallable, Category = "LLMAI|Voice")
void SetupAudioPlayback(UAudioComponent* AudioComponent, 
                            bool bIgnoreFlushing = true);

Audio Streaming Integration​

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio Stream")
void SetAudioStreamComponent(ULLMAIAudioStreamComponent* AudioComponent);

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio Stream")
void StartAudioStreaming();

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio Stream")
void StopAudioStreaming();

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio Stream", BlueprintPure)
bool IsAudioStreaming() const;

Auto-Creation Status​

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio Stream", BlueprintPure)
bool IsAudioStreamAutoCreated() const;

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio Stream", BlueprintPure)
bool IsAudioPlaybackAutoCreated() const;

UFUNCTION(BlueprintCallable, Category = "LLMAI|Voice", BlueprintPure)
bool IsAudioPlaying() const;

Response & Session Identifiers​

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket", BlueprintPure)
FString GetCurrentSessionIdentifier() const;

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket", BlueprintPure)
FString GetCurrentResponseId() const;