Run AI Models

Getting Started

Enable AI in your NuxtHub project by adding the ai property to the hub object in your nuxt.config.ts file.

export default defineNuxtConfig({
  hub: {
    ai: true
  },
})

hubAI()

Server composable that returns a Workers AI client.

const ai = hubAI()

run()

Runs a model. Takes a model as the first parameter, and an object as the second parameter.

export default defineEventHandler(async () => {
  const ai = hubAI() // access AI bindings
  return await ai.run('@cf/meta/llama-3.1-8b-instruct', {
    prompt: 'Who is the author of Nuxt?'
  })
})

Options

Models

Workers AI comes with a curated set of popular open-source models that enable you to do tasks such as image classification, text generation, object detection and more.

Streaming

The recommended method to handle text generation responses is streaming.

LLMs work internally by generating responses sequentially using a process of repeated inference — the full output of a LLM model is essentially a sequence of hundreds or thousands of individual prediction tasks. For this reason, while it only takes a few milliseconds to generate a single token, generating the full response takes longer, on the order of seconds.

You can use streaming to start displaying the response as soon as the first tokens are generated, and append each additional token until the response is complete. This yields a much better experience for the end user. Displaying text incrementally as it’s generated not only provides instant responsiveness, but also gives the end-user time to read and interpret the text.

To enable, set the stream parameter to true.

You can check if the model you're using support streaming on Cloudflare's models documentation.

export default defineEventHandler(async (event) => {
  const messages = [
    { role: 'system', content: 'You are a friendly assistant' },
    { role: 'user', content: 'What is the origin of the phrase Hello, World' }
  ]

  const ai = hubAI()
  const stream = await ai.run('@cf/meta/llama-3.1-8b-instruct', {
    stream: true,
    messages
  })
  return sendStream(event, stream)
})

AI Gateway

Workers AI is compatible with AI Gateway, which enables caching responses, analytics, real-time logging, ratelimiting, and fallback providers. Learn more about AI Gateway.

Options

Configure options for AI Gateway by passing an additional object to hubAI().run(), learn more on Cloudflare's docs.

export default defineEventHandler(async () => {
  const ai = hubAI()
  return await ai.run('@cf/meta/llama-3-8b-instruct',
    {
      prompt: 'Who is the creator of Nuxt?'
    },
    {
      gateway: {
        id: '{gateway_slug}',
        skipCache: false,
        cacheTtl: 3360
      }
    })
})

Templates

Explore open source templates made by the community:

Vercel AI SDK

It is possible to use the Vercel AI SDK with Cloudflare Workers AI.

NuxtHub AI is compatible with some functions of the Vercel AI SDK, which enables streaming responses.

Make sure to install the Vercel AI SDK in your project.

npx nypm i ai @ai-sdk/vue

useChat()

To leverage the useChat() Vue composable, you need to create a POST /api/chat endpoint that uses the hubAI() server composable and returns a compatible stream for the Vercel AI SDK.

import { AIStream, formatStreamPart } from 'ai'

export default defineEventHandler(async (event) => {
  const { messages } = await readBody(event)

  const stream = await hubAI().run('@cf/meta/llama-3.1-8b-instruct', {
    messages,
    stream: true
  }) as ReadableStream

  // Return a compatible stream for the Vercel AI SDK
  return AIStream(
    new Response(stream),
    data => formatStreamPart('text', JSON.parse(data).response)
  )
})

Then, we can create a chat component that uses the useChat() composable.

<script setup lang="ts">
import { useChat } from '@ai-sdk/vue'

const { messages, input, handleSubmit, isLoading, stop, error, reload } = useChat()
</script>

<template>
  <div v-for="m in messages" :key="m.id">
    {{ m.role }}: {{ m.content }}
  </div>
  <div v-if="error">
    <div>{{ error.message || 'An error occurred' }}</div>
    <button @click="reload">retry</button>
  </div>
  <form @submit="handleSubmit">
    <input v-model="input" placeholder="Type here..." />
    <button v-if="isLoading" @click="stop">stop</button>
    <button v-else type="submit">send</button>
  </form>
</template>

Learn more about the useChat() Vue composable.