In this article we will explore how to get authentication working in your Next.js app using Auth.js and Xata. It's important to note that Next.js has at the moment 2 routing architectures that are defined via your project's file system:

- App: where routes are mapped in `/app` directory
- Pages: where routes are mapped in `/pages` directory

Working the conditions and conventions of each directory can get you with any routes landscape. Though there are fundamental differences on how you will think the data story depending on which architecture you choose. If you've been working (or has worked) with Next.js at any point until now, chances are you're familiar with the pages directory architecture. And in case you haven't exactly kept track to the new experimental releases since Next.js 13, there will be some catch-up to do before you start using and [understanding the App Directory](https://www.smashingmagazine.com/2023/02/understanding-app-directory-architecture-next-js/).

Before going into the Next.js specifics, let's first understand our goals with this project. Visualizing the mental model is an important step to ensure your user's data is switching hands as little as possible.

## Defining a Mental Model

A diagram is probably the fastest way to visualizing a process. This is important to see if there aren't any redundant calls, or if any of those requests are chained in some condition. Chained requests are the grim reapers of perceived performance, we need to get those out of the way as soon as possible.

![diagram mapping how authentication will work in our app](/images/blog/auth_js/diagram.png)

In the above graph, we have “Frontend” and “Backend” that are both handled by Next.js. We have the “auth provider” which our app is connected to via Auth.js (within a Next.js endpoint), and we have Xata as our database, connecting as well via Next.js’ backend layer. We can examine and translate the flow of that diagram into 3 steps:

1. **Authentication**
Provider performs user authentication, the library creates a `session` and the app receives that as a `payload` from the auth request.
2. **Resource Request**
Authenticated User performs request with `resourceId`, app takes `userId` from `session`
3. **Granting Access**
Xata filters all resources from table to only the ones owned by `userId` and returns (if exists) the one with `resourceId`

Now that we aligned ourselves about the target implementation, it's time to roll up our sleeves and start writing some code. First, let's prepare the authentication endpoint.

## Setup Auth

When working with Auth.js and Next.js, everything starts with a very special API Route. We create a dynamic catch-all route: `api/[...nextauth].ts`. There's a lot to unpack on this naming pattern for first timers, let's have a quick look:

- Everything inside `/pages/api/auth` is a resource route in Next.js, it will either be a Serverless Function or an Edge Function depending on the runtime you choose.
- Filenames surrounded by square brackets (`[]`) indicate this route is dynamic, so it will take the corresponding segment path and add to the search params of the route. E.g.: `[slug].js` will serve `[serverless.com/xata](<http://serverless.com/xata>)` and receive `{ slug: "xata" }`as a parameter.
- By adding the spread (`…`) to the param, we indicate we take all segment paths. So not only `/api/xata` but also `/api/xata/auth/another/route` will be rendered by that route. Hence why we namespace our catch-all route inside `/auth`

> There is currently an [opened issue within the Auth.js repository](https://github.com/nextauthjs/next-auth/issues/6792) to support the new [Route Handlers](https://beta.nextjs.org/docs/routing/route-handlers), if your intention is going full `/app` directory, keep an eye on it. Otherwise, everything will work just fine keeping the API Route inside `/pages`.

Our Serverless Route will then import `NextAuth` as a high order function, this will handle the necessary routes the library will need to implement and we will only pass the configuration object to it. Xata has an adapter, which means once the authentication is done, Auth.js will itself create an entry into a Xata database for that user, so the authentication provider will only be used for registering the user and sign-in. Auth.js has a big number of authentication providers, for this example we will use GitHub. Your `/pages/api/auth/[...nextauth].ts` will then look like this:

```tsx
import NextAuth from 'next-auth'
import GitHubProvider from 'next-auth/providers/github'
import { XataAdapter } from '@next-auth/xata-adapter'
import { XataClient } from '~/shared/xata.codegen'

const client = new XataClient()

export const authConfig = {
  adapter: XataAdapter(client),
  providers: [
    GitHubProvider({
      clientId: process.env.GITHUB_ID,
      clientSecret: process.env.GITHUB_SECRET,
    }),
  ],
}

export default NextAuth(authConfig)

```

With that, Auth.js is ready to authenticate and provide our app with a valid session to fetch user specific data. Now it's time to setup our database our app can communicate to Xata, it is important to setup a workspace and push our schema to it. First ,take the following `json` and save it somewhere:

```json
{
  "tables": [
    {
      "name": "nextauth_users",
      "columns": [
        {
          "name": "email",
          "type": "email"
        },
        {
          "name": "emailVerified",
          "type": "datetime"
        },
        {
          "name": "name",
          "type": "string"
        },
        {
          "name": "image",
          "type": "string"
        }
      ]
    },
    {
      "name": "nextauth_accounts",
      "columns": [
        {
          "name": "user",
          "type": "link",
          "link": {
            "table": "nextauth_users"
          }
        },
        {
          "name": "type",
          "type": "string"
        },
        {
          "name": "provider",
          "type": "string"
        },
        {
          "name": "providerAccountId",
          "type": "string"
        },
        {
          "name": "refresh_token",
          "type": "string"
        },
        {
          "name": "access_token",
          "type": "string"
        },
        {
          "name": "expires_at",
          "type": "int"
        },
        {
          "name": "token_type",
          "type": "string"
        },
        {
          "name": "scope",
          "type": "string"
        },
        {
          "name": "id_token",
          "type": "text"
        },
        {
          "name": "session_state",
          "type": "string"
        }
      ]
    },
    {
      "name": "nextauth_verificationTokens",
      "columns": [
        {
          "name": "identifier",
          "type": "string"
        },
        {
          "name": "token",
          "type": "string"
        },
        {
          "name": "expires",
          "type": "datetime"
        }
      ]
    },
    {
      "name": "nextauth_users_accounts",
      "columns": [
        {
          "name": "user",
          "type": "link",
          "link": {
            "table": "nextauth_users"
          }
        },
        {
          "name": "account",
          "type": "link",
          "link": {
            "table": "nextauth_accounts"
          }
        }
      ]
    },
    {
      "name": "nextauth_users_sessions",
      "columns": [
        {
          "name": "user",
          "type": "link",
          "link": {
            "table": "nextauth_users"
          }
        },
        {
          "name": "session",
          "type": "link",
          "link": {
            "table": "nextauth_sessions"
          }
        }
      ]
    },
    {
      "name": "nextauth_sessions",
      "columns": [
        {
          "name": "sessionToken",
          "type": "string"
        },
        {
          "name": "expires",
          "type": "datetime"
        },
        {
          "name": "user",
          "type": "link",
          "link": {
            "table": "nextauth_users"
          }
        }
      ]
    }
  ]
}

```

And now if you have our CLI installed, you can use the  `xata` namespace, otherwise `npx @xata.io/cli@latest` will also work:

```bash
xata init --schema="path/to/schema.json"

```

Finally, all code is in place to:

1. Connect to GitHub OAuth
2. Create a session within our Next.js app
3. Push user registration to our Xata database

What we need now is a set of keys to effectively connect to those services in a secure way.

## Setup Environment Variables

The environment variables `GITHUB_ID` and `GITHUB_SECRET` are provided by GitHub once you register an [OAuth App](https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/creating-an-oauth-app). For your code to run properly you will need to have a `.env` (or `.env.local`) file looking like:

```
# Xata specifics
XATA_API_KEY=<Xata API Key>
XATA_BRANCH=main

# GitHub OAuth specifics
GITHUB_ID=<id of your Provider OAuth app for this environment>
GITHUB_SECRET=<secret of your Provider OAuth app for this environment>

# Auth.js specifics
NEXTAUTH_URL=http://localhost:3000 # if in local dev
NEXTAUTH_SECRET=<a random secret string>

```

Finally, it's time to manage some data.

## Fetching User Data

There are a few ways with which you can provide secure access to routes in your Auth.js app. One is via the Auth.js middleware, this will redirect anyone without a `session` automatically to the login route you set up. This approach should work in both App Directory and Pages Directory.

If working with the Pages Directory, you must set the `<SessionProvider>` encapsulating routes that should have access to the Session (if using `getStaticProps`), or check the session within `getServerSideProps` and pass it down as render props.

Lastly, in the app directory, the best way is to use `getServerSession` method, which will retrieve the method in every Server Component that needs to use it. This is close to what `getServerSideProps` can do, though it's on a component level - while on `getServerSideProps`we're dealing at route level.

When working with React Server components, don't worry about firing multiple requests, it always hit the same endpoint and therefore Next.js will be able to deduplicate it, so there will only be one fetch request.

## Accessing User Specific Data

Once we got Auth setup, we can treat Row-Level Security (RLS) as an if/else statement. For example, take this contrived profile page:

```tsx
import { fetchUserData } from '~/lib/db.server.ts'
import { getSession } from 'next-auth/react'

export default function Profile({ user, userData }) {
	return ( /** React component **/ )
}

export const getServerSideProps = async ({
  req,
  res,
}) => {
  const session = await getSession({ req })

  return {
    props: {
      user: session?.user,
      todos: await fetchUserData(session?.user.email),
    },
  }
}

```

In the above example, we are using the user's email as their unique identifier (or Primary Key). And we implemented a getter in a file named `db.server.ts`. Let's see how that looks like using the Xata SDK.

```tsx
// the file generated via our CLI
import { getXataClient } from '~/lib/xata.codegen.server'

const xata = getXataClient()

export const fetchUserData = async (email: string) => {
  const userData = await xata.db.itemList
    .filter({
      user: {
        email,
      },
    })
    .sort('created_at', 'desc')
    .getPaginated({ pagination: { size: 100, offset: 0 }})

  return userData
}

```

Remember, this method may **only** be accessed from the server-side. In Next.js this means:

- from a React Server Component
- from `getStaticProps`, `getStaticPaths`, or `getServerSideProps`
- from a Middleware, Serverless Function, or Edge Function

To retrieve data from the client-side, we would need a different method that sends a `POST` request to an API Route or a Route Handler, for example:

```jsx
export const fetchData = async (userEmail: string) => {
  const response = await fetch('/api/fetch-data', {
    method: 'post',
    body: JSON.stringify({
      userEmail,
    }),
  })

  if (response.ok) {
    return response
  } else {
    const err = await response.json()
    throw new Error(
      `ERROR [${response.status}]: ${
        err.message ?? 'The server returned an error'
      }`
    )
  }
}

```

## Working Code

Now that we are done, you can compare your code or check the components that we wrote to use this logic in our examples repository:

- [App Directory example](https://github.com/xataio/examples/tree/main/apps/sample-next-auth-app).
- [Pages Directory example](https://github.com/xataio/examples/tree/main/apps/sample-next-auth-pages).

## Conclusion

I hope this article has helped you getting started with Row-Level Security and got you through the door with using Auth in a Next.js app with no vendor lock-in and owning your data. If any questions remain, reach Atila both on our [discord](https://xata.io/discord) or on [twitter](https://twitter.com/intent/follow?screen_name=atilafassina).

Let's build Next.js apps with JWT authentication, with Auth.js and Xata. With app directory and React Server Components, or with the traditional pages routing system.

Xata and Auth.js on 2 Next.js apps: App Directory and Pages Directory

At the same time we launched our [ChatGPT integration](https://xata.io/chatgpt), we also added a new vector type to Xata, making it easy to store embeddings. Additionally, we have added a new `vectorSearch` endpoint, which performs similarity search on embeddings.

Let’s take a quick tour to see how you can use these new capabilities to implement semantic search. We’re going to use the OpenAI embeddings API, TypeScript and Deno. This tutorial assumes prior knowledge of TypeScript, but no prior knowledge of Xata, Deno, or OpenAI.

## What is semantic search?

Instead of just matching keywords, as traditional search engines do, semantic search attempts to understand the context and intent of the query and the relationships between the words used in it.

For example, let’s say you have the following sample sentences:

- "The quick brown fox jumps over the lazy dog story"
- “The vehicle collided with the tree”
- “The cat growled towards the dog”
- "Lorem ipsum dolor sit amet, consectetur adipiscing elit”
- “The sunset painted the sky with vibrant colors”

If you search for “sample text in latin”, traditional keyword search won’t match the “lorem ipsum” text, but semantic search will (and we’re going to demo it in this article).

Similarly, if you search for “the kitty hissed at the puppy”, semantic search will see that the phrase has the same meaning as “the cat growled towards the dog”, even if they use none of the same words. Or, for another example, “vanilla sky” should bring up the “The sunset painted the sky with vibrant colors” sentence. Pretty cool, right? This is now quite possible thanks large-language models and vector search.

## A quick intro to embeddings

From a data point of view, embeddings are simply arrays of floats. They are the output of ML models, where the input can be a word, a sentence, a paragraph of text, an image, an audio file, a video file, and so on.

![/images/blog/semantic-search/embeddings_diagram.png](/images/blog/semantic-search/embeddings_diagram.png)

Each number in the array of floats represents the input text on a particular dimension, which depends on the model. This means that the more similar the embeddings are, the more “similar” the input objects are. I’ve put “similar” in quotes because it depends on the model what kind of similar it means. When it comes to text, it’s usually about “similar in meaning”, even if different words, expressions, or languages are used.

Reducing complex data to an array of numbers representing its qualities turns out to be very useful for a number of use cases. Think of reverse image search, recommendations algorithms for video and music, product recommendations, categorizing products, and so on.

## Vector type in Xata

If you want to follow along, start with steps:

* Sign up or sign into Xata [here](https://app.xata.io/signin) (the usage from this tutorial fits well within the free tier, so you don’t need to set up billing)
* Create a database named `vectors`
* Create a table named `Sentences`
* Add two columns:
   * `sentence` of type string
   * `embedding` of type vector. Use 1536 as the dimension

![/images/blog/semantic-search/add_vector_type_screenshot.png](/images/blog/semantic-search/add_vector_type_screenshot.png)

When you are done, the schema should look like this:

![/images/blog/semantic-search/view_schema_screenshot.png](/images/blog/semantic-search/view_schema_screenshot.png)

## Initialize the Xata project

To get ready for running the typescript code, install the Xata CLI:

```sh
npm install -g @xata.io/cli@latest

```

Run `xata auth login` to authenticate the CLI. This will open a browser window and prompt you to generate a new API key. Give it any name you’d like.

```sh
xata auth login

```

Create a folder for the code:

```sh
mkdir sentences
cd sentences

```

And run `xata init` to connect it to the Xata DB:

```sh
xata init

```

The Xata CLI will ask you to select the database and then ask you how to generate the types. Select **Generate TypeScript code with Deno imports**. Use default settings for the rest of the questions.

![/images/blog/semantic-search/xata_init_install_screenshot.png](/images/blog/semantic-search/xata_init_install_screenshot.png)

## Prepare OpenAI and Deno

Create an [OpenAI account](https://platform.openai.com/signup) and generate a key. Note that you need to set up billing for OpenAI in order for to run these examples, but the cost will be tiny (under $1).

Add the OpenAI key to the `.env` file which was created by the `xata init` command above. Your `.env` should look something like this:

```sh
# API key used by the CLI and the SDK
# Make sure your framework/tooling loads this file on startup to have it available for the SDK
XATA_API_KEY=xau_<redacted>
OPENAI_API_KEY=sk-<redacted>

```

Install the Deno CLI. See [this page](https://deno.land/manual@v1.31.3/getting_started/installation) for the various install options, on macOs with Homebrew it is:

```sh
brew install deno

```

## Load data

It’s now the time to write a bit of TypeScript code. Create a `loadWithEmbeddings.ts` file top level in your project with the following contents:

```ts
import { Configuration, OpenAIApi } from "npm:openai";
import { getXataClient } from "./src/xata.ts";
import { config as dotenvConfig } from "<https://deno.land/x/dotenv@v1.0.1/mod.ts>";

dotenvConfig({ export: true });

const openAIConfig = new Configuration({
  apiKey: Deno.env.get("OPENAI_API_KEY"),
});
const openAI = new OpenAIApi(openAIConfig);
const xata = getXataClient();

const sentences: string[] = [
  "The quick brown fox jumps over the lazy dog story",
  "The vehicle collided with the tree",
  "The cat growled towards the dog",
  "Lorem ipsum dolor sit amet, consectetur adipiscing elit",
  "The sunset painted the sky with vibrant colors",
];

for (const sentence of sentences) {
  const resp = await openAI.createEmbedding({
    input: sentence,
    model: "text-embedding-ada-002",
  });
  const [{ embedding }] = resp.data.data;

  xata.db.Sentences.create({
    sentence,
    embedding,
  });
}

```

Step by step, this is what the script is doing:

- Uses `dotenv` to load the `.env` file into the current environment. This makes sure the `XATA_API_KEY` and the `OPENAI_API_KEY` are available to the rest of the script.
- Initializes the OpenAI and Xata client libraries.
- Defines the test data in the `sentences` array.
- For each `sentence` , calls the [OpenAI embeddings API](https://platform.openai.com/docs/guides/embeddings) to get the embedding for it.
- Inserts a record containing the sentence and the embedding into Xata.

Execute the script like this:

```sh
deno run --allow-net --allow-env --allow-read --allow-run ./loadWithEmbeddings.ts

```

If you visit the Xata UI now, you should see the data loaded, together with the embeddings.

![/images/blog/semantic-search/view_data_screenshot.png](/images/blog/semantic-search/view_data_screenshot.png)

## Run search queries

Let’s write another simple script that performs a search based on an input query. Name this script `search.ts`.

```ts
import { Configuration, OpenAIApi } from "npm:openai";
import { getXataClient } from "./src/xata.ts";
import { config as dotenvConfig } from "<https://deno.land/x/dotenv@v1.0.1/mod.ts>";

dotenvConfig({ export: true });

const openAIConfig = new Configuration({
  apiKey: Deno.env.get("OPENAI_API_KEY"),
});
const openAI = new OpenAIApi(openAIConfig);
const xata = getXataClient();

if (Deno.args.length !== 1) {
  console.log("Please provide a search query");
  console.log(
    "Example: deno run --allow-net --allow-env search.ts 'the quick brown fox'"
  );
  Deno.exit(1);
}

const query = Deno.args[0];
const resp = await openAI.createEmbedding({
  input: query,
  model: "text-embedding-ada-002",
});
const [{ embedding }] = resp.data.data;

const results = await xata.db.Sentences.vectorSearch("embedding", embedding);

for (const result of results) {
  console.log(result.getMetadata().score, "|", result.sentence);
}

```

Here is what’s going on in the script:

- The beginning is similar as for the previous script, using `dotenv` to load the `.env` file and then initializing the client libraries for OpenAI and Xata
- Read the query as the first argument passed to the script
- Use the [OpenAI embeddings API](https://platform.openai.com/docs/guides/embeddings) to create embeddings for the query
- Run a vector search using the Xata [Vector Search API.](https://xata.io/docs/typescript-client/vector-search) This finds vectors that are similar to the provided embedding
- Print the results, together with the similarity score

To run the script, execute it like this:

```sh
$ deno run --allow-net --allow-env --allow-read --allow-run \\
  ./search.ts 'sample text in latin'

1.8154079 | Lorem ipsum dolor sit amet, consectetur adipiscing elit
1.7424928 | The quick brown fox jumps over the lazy dog story
1.7360129 | The cat growled towards the dog
1.7311659 | The sunset painted the sky with vibrant colors
1.7038174 | The vehicle collided with the tree

```

As you can see, searching for “sample text in latin” results in the “Lorem ipsum” text as we hoped. You can also try some variations, for example, “sample sentence” still brings the “Lorem ipsum” one as the top result:

```sh
$ deno run --allow-net --allow-env --allow-read --allow-run \\
  ./search.ts 'sample sentence'

1.805396 | Lorem ipsum dolor sit amet, consectetur adipiscing elit
1.7715557 | The quick brown fox jumps over the lazy dog story
1.7608802 | The sunset painted the sky with vibrant colors
1.7573793 | The cat growled towards the dog
1.7493906 | The vehicle collided with the tree

```

The scores on the left are numbers between 0 and 2 which indicate how close each sentence is to the provided query. If you run with a sentence that exits in the data, you’ll get a score very close to 2:

```sh
$ deno run --allow-net --allow-env --allow-read --allow-run \\
  ./search.ts 'The quick brown fox jumps over the lazy dog story'

1.9999993 | The quick brown fox jumps over the lazy dog story
1.8063612 | The cat growled towards the dog
1.769694 | Lorem ipsum dolor sit amet, consectetur adipiscing elit
1.7673006 | The vehicle collided with the tree
1.7605586 | The sunset painted the sky with vibrant colors

```

Now let’s try the “vanilla sky” query:

```sh
$ deno run --allow-net --allow-env --allow-read --allow-run \\
  ./search.ts 'vanilla sky'

1.8137007 | The sunset painted the sky with vibrant colors
1.7584264 | Lorem ipsum dolor sit amet, consectetur adipiscing elit
1.7579571 | The quick brown fox jumps over the lazy dog story
1.7505519 | The vehicle collided with the tree
1.738231 | The cat growled towards the dog

```

Bingo, top result matches what we expected.

Another one to try:

```sh
$ deno run --allow-net --allow-env --allow-read --allow-run \\
  ./search.ts 'The car crashed into the oak.'

1.907266 | The vehicle collided with the tree
1.7763984 | The cat growled towards the dog
1.7755028 | The quick brown fox jumps over the lazy dog story
1.7745116 | The sunset painted the sky with vibrant colors
1.7570261 | Lorem ipsum dolor sit amet, consectetur adipiscing elit

```

Again, the sentence with the same meaning shows up at the top with a score close to 2.

## Conclusion

The large-language-models are powerful tools that open up new use cases. Semantic search is one of these use cases, and we’ve seen how the Xata vector search can be used to implement it. You can also use it to build recommendation engines, or finding similar entries in a knowledge-base, or questions in a Q&A website.

If you’re running this tutorial, please join us on the Xata [Discord](https://xata.io/discord) and let us know what you are building!

A tutorial showing how to implement semantic search, using Xata, the OpenAI embeddings API, TypeScript, and Deno.

Semantic search with Xata, OpenAI, TypeScript, and Deno

At Xata we always look for ways to improve our performance and user experience. One way to achieve this is by reducing the latency of our API responses. In this post, we will share our experience migrating to Cloudflare and the benefits we gained.

First, let's talk about Time to First Byte (TTFB) – the time it takes for the first byte of the response to be sent back to the user. TTFB is affected by many factors, including network latency, server processing time, and the time it takes for the DNS lookup. As a general rule of thumb, most sites should strive to have a TTFB of 0.8 seconds or less.

You can’t improve what you don’t measure, so let’s see what Xata's response times were when we started working on this:

```bash
$ curl -so /dev/null https://us-east-1.xata.sh -w "| dnslookup: %{time_namelookup} \n| connect: %{time_connect} \n| appconnect: %{time_appconnect} \n| pretransfer: %{time_pretransfer} \n| starttransfer: %{time_starttransfer} \n| total: %{time_total} \n| size: %{size_download}\n"

| dnslookup: 0.144752
| connect: 0.254816
| appconnect: 0.493482
| pretransfer: 0.493587
| starttransfer: 0.610611
| total: 0.610689
| size: 110
```

We made further tests to have a clear picture of the current state. We’ve also used [Vercel’s edge data latency app](https://edge-data-latency.vercel.app/). The app performs simple queries to a specific Xata database table in two different ways:

Regional: sends 10 requests to a Vercel function that is pinned to be executed in the N. Virginia region (us-east-1). These are called “regional” requests.

Global: sends 10 requests to a Vercel function that is not pinned to a region, it picks a region close to where we run the Vercel function from – in our case, Spain (Europe).

![latency_distribution1](/images/blog/latency/latency_dist1.png)

“Global” response times are worse than “regional” response times, but digging deeper, we saw that by consistently sending requests the times decreased, but not as much as we expected. This prompted us to seek improvement, and we conducted a proof of concept to test Cloudflare’s services.

Cloudflare is a content delivery network (CDN) that provides various services designed to improve the performance and security of websites and applications. It operates a global network of servers in over 285 cities spread across the globe, ensuring that they are as close to end-users as possible.

We put Cloudflare in front of our API, and let it handle DNS resolution and TLS termination happen closer to the user. Cloudflare's DNS resolver is fast and reliable, and it caches DNS records to ensure that subsequent requests are even faster. We observed some improvement in DNS lookup times by using Cloudflare's globally distributed DNS network, from ~150ms without Cloudflare to ~10ms using Cloudflare.

When a user makes a request to our API, Cloudflare's DNS resolver directs the request to the nearest Cloudflare server. This means that the request doesn't have to travel as far, resulting in lower latency and faster TTFB times. Cloudflare's Anycast network can also further reduce latency by routing traffic to the nearest server with the lowest latency, and it uses smart routing algorithms to optimize network paths.

During the proof of concept, we also tried Cloudflare Argo Smart Routing. Argo is a service that optimizes content delivery over the Cloudflare network, using a combination of smart routing and protocol optimizations. The results showed a slight improvement in response times when using Argo, but it was not significant for our use case.

We conducted load tests to compare different configurations sending requests for 120 seconds without reusing the same connection per each request, the resume of the results is shown below:

![load_test_table](/images/blog/latency/load_test_table.png)

To further validate our findings, we also executed several runs of our end-to-end (E2E) tests to gather more data on the performance difference. On one side, we ran our usual tests with our default configuration, on the other side we configured a secondary domain in Cloudflare and proxied the requests to our API. The results showed that using Cloudflare configuration significantly reduced the duration of our tests. We saved on average about 60 seconds (>25%) in Xata eu-west-1 region, as shown below:

```
Before
Test Files 15 passed (15) Tests 172 passed | 2 skipped (174) Start at 11:07:13 Duration 221.77s (transform 793ms, setup 2ms, collect 7.42s, tests 203.69s)

Now
Test Files 15 passed (15) Tests 172 passed | 2 skipped (174) Start at 08:32:56 Duration 157.63s (transform 822ms, setup 2ms, collect 7.09s, tests 141.18s)
```

We considered after this PoC that the benefits Cloudflare brought us were enough to make the change, so we migrated our production DNS to Cloudflare and switched our traffic to the Cloudflare network to proxy the requests to our API.

### Conclusions

We have successfully decreased our API response times by leveraging Cloudflare's DNS and proxying the traffic through the Cloudflare network.

Our load testing results demonstrate that using Cloudflare in front of our API allowed us to improve our overall performance. Prior to using Cloudflare, our response times from our load test result were not able to go below 360ms. However, with Cloudflare, we achieved faster response times consistently, leading to a better user experience.

![latency_distribution2](/images/blog/latency/latency_dist2.png)

Benefits:

- Cloudflare's globally distributed DNS network improves DNS lookup times.
- Cloudflare can provide an appreciable improvement in response times when there is consistent traffic routed to our servers.
- Cloudflare offers us different features that we can activate eventually, such as Argo Smart Routing. Argo optimizes network transit time, can reduce Internet latency on average by 30%, and can dynamically select the best path for each request based on factors such as network congestion, latency, and available capacity.

Drawbacks:

- Cloudflare does not always improve Time To First Byte (TTFB) of cold connections, especially when traffic is low.
- When the request originates near the servers, the improvement in response time may not be noticeable.

Xata already uses Cloudflare Workers for Platform to host [our serverless functions](https://xata.io/blog/xata-workers) and some new exciting features coming soon to Xata will also use Cloudflare services. Ready to experience our fast response times and start building something amazing on Xata? [Sign up](https://app.xata.io/signup) now and join our community on [Discord](https://xata.io/discord). We can’t wait to see what you build!

In this blog we share our experience migrating to Cloudflare and the reduction in latency we gained by making the switch.

How we reduced our API latency by 50% using Cloudflare

Last week we’ve added a Q&A bot that answers questions from [our documentation](https://xata.io/docs/overview?show-chat). This leverages the ChatGPT tech to answer questions from the Xata documentation, even though the OpenAI GPT model was never trained on the Xata docs. 

![GIF of the Xata bot answering a question](/images/blog/keyword_or_vector/xata-bot-preview.gif)

The way we do this is by using an approach suggested by Simon Willison in this [blog post](https://simonwillison.net/2023/Jan/13/semantic-search-answers/). The same approach can be found also in an [OpenAI cookbook](https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb). The idea is the following:

- Run a text search against the documentation to find the content that is most relevant to the question asked by the user.
- Produce a prompt with this general form:

```
With these rules: {rules}
And this text: {context}
Given the above text, answer the question: {question}
Answer:
```

- Send the prompt to the ChatGPT API and let the model complete the answer.

We found out that this works quite well and, combined with a relatively low model temperature (the concept of temperature is explained in this [blog post](https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/)), this tends to produce correct results and code snippets, as long as the answer can be found in the documentation.

A key limitation to this approach is that the prompt that you build in the second step above needs to have max 4000 tokens (~3000 words). This means that the first step, the text search to select the most relevant documents, becomes really important. If the search step does a good job and provides the right context, ChatGPT tends to also do a good job in producing a correct and to-the-point result.

So what’s the best way to find the most relevant pieces of content in the documentation? The OpenAI cookbook, as well as Simon’s blog, use what is called semantic search. Semantic search leverages the language model to generate embeddings for both the question and the content. Embeddings are arrays of numbers that represent the text on a number of dimension. Pieces of text that have similar embeddings have a similar meaning. This means a good strategy is to find the pieces of content that the most similar embeddings to the question embeddings.

Another possible strategy, based on the more classical keyword search, looks like this:

- Ask ChatGPT to extract the keywords from the question, with a prompt like this:

```
Extract keywords for a search query from the text provided. 
Add synonyms for words where a more common one exists.
```

- Use the provided keywords to run a free-text-search and pick the top results

Putting it in a single diagram, the two methods look like this:

![Diagram showing how we use keyword or vector search with ChatGPT](/images/blog/keyword_or_vector/xata-chatgpt-diagram.png)

We have tried both on our documentation and have noticed some pros and cons. 

Let’s start by comparing a few results. Both are ran against the same database, and they both use the ChatGPT `gpt-3.5-turbo` model. As there is randomness involved, I ran each question 2-3 times and picked what looked to me like the best result.

## Question: How do I install the Xata CLI?

**Answer with vector search:**

![Question for the Xata bot: How do I install the Xata CLI](/images/blog/keyword_or_vector/install_question_vector.png)

**Answer with keyword search:**

![Question for the Xata bot: How do I install the Xata CLI](/images/blog/keyword_or_vector/install_question_keyword.png)

*Verdict*: Both versions provided the correct answer, however the vector search one is a bit more complete. They both found the correct docs page for it, but I think our highlights-based heuristic selected a shorter chunk of text in case of the keyword strategy. Winner: vector search. 

Score: 1-0

## Question: How do you use Xata with Deno?

**Answer with vector search:**

![Question for the Xata bot: How do you use Xata with Deno?](/images/blog/keyword_or_vector/deno_question_vector.png)

**Answer with keyword search:**

![Question for the Xata bot: How do you use Xata with Deno?](/images/blog/keyword_or_vector/deno_question_keyword.png)

**Verdict:** Disappointing result for vector search, who somehow missed the dedicated Deno page in our docs. It did find some other Deno relevant content, but not the page that contained the very useful example. Winner: keyword search.

Score: 1-1

## Question: How can I import a CSV file with custom column types?

**With vector search:**

![Question for the Xata bot: How can I import a CSV file with custom column types?](/images/blog/keyword_or_vector/import_csv_question_vector.png)

**With keyword search:**

![Question for the Xata bot: How can I import a CSV file with custom column types?](/images/blog/keyword_or_vector/import_csv_question_keyword.png)

**Verdict:** Both have found the right page (”Import a CSV file”), but the keyword search version managed to get a more complete answer. I did run this multiple times to make sure it’s not a fluke. I think the difference comes from how the text fragment is selected (neighbouring the keywords in case of keyword search, from the beginning of the page in case of vector search). Winner: keyword search.

Score: 1-2

## Question: How can I filter a table named Users by the email column?

**With vector search:**

![Question for the Xata bot: How can I filter a table named Users by the email column?](/images/blog/keyword_or_vector/filter_question_vector.png)

**With keyword search:**

![Question for the Xata bot: How can I filter a table named Users by the email column?](/images/blog/keyword_or_vector/filter_question_keyword.png)

*Verdict:* The vector search did better on this one, because it found the “Filtering” page on which there were more examples that ChatGPT could use to compose the answer. The keyword search answer is subtly broken, because it uses “query” instead of “filter” for the method name. Winner: vector search.

Score: 2-2

## Question: What is Xata?

**With vector search:**

![Question for the Xata bot: What is Xata?](/images/blog/keyword_or_vector/what_is_xata_vector.png)

**With keyword search:**

![Question for the Xata bot: What is Xata?](/images/blog/keyword_or_vector/what_is_xata_keyword.png)

**Verdict:** This one is a draw, because both answers are quite good. The two picked different pages to summarize in an answer, but both did a good job and I can’t pick a winner.

Score: 3-3

## Configuration and tuning

This is a sample Xata request used for keyword search:

```json
// POST https://workspace-id.eu-west-1.xata.sh/db/docs:main/tables/search/ask
{
	"question": "What is Xata?",
	"rules": [
		"Do not answer questions about pricing or the free tier. Respond that Xata has several options available, please check https://xata.io/pricing for more information.",
		"If the user asks a how-to question, provide a code snippet in the language they asked for with TypeScript as the default.",
		"Only answer questions that are relating to the defined context or are general technical questions. If asked about a question outside of the context, you can respond with \"It doesn't look like I have enough information to answer that. Check the documentation or contact support.\"",
		"Results should be relevant to the context provided and match what is expected for a cloud database.",
		"If the question doesn't appear to be answerable from the context provided, but seems to be a question about TypeScript, Javascript, or REST APIs, you may answer from outside of the provided context.",
	"If you answer with Markdown snippets, prefer the GitHub flavour.",
	"Your name is DanGPT"
	],
	"searchType": "keyword",
	"search": {
		"fuzziness": 1,
		"target": [
			"slug",
			{
				"column": "title",
				"weight": 4
			},
			"content",
			"section",
			{
				"column": "keywords",
				"weight": 4
			}
		],
		"boosters": [
			{
				"valueBooster": {
					"column": "section",
					"value": "guide",
					"factor": 18
				}
			}
		]
	}
}
```

And this what we use for vector search:

```json
// POST https://workspace-id.eu-west-1.xata.sh/db/docs:main/tables/search/ask
{
	"question": "How do I get a record by id?",
	"rules": [
		"Do not answer questions about pricing or the free tier. Respond that Xata has several options available, please check https://xata.io/pricing for more information.",
		"If the user asks a how-to question, provide a code snippet in the language they asked for with TypeScript as the default.",
		"Only answer questions that are relating to the defined context or are general technical questions. If asked about a question outside of the context, you can respond with \"It doesn't look like I have enough information to answer that. Check the documentation or contact support.\"",
		"Results should be relevant to the context provided and match what is expected for a cloud database.",
		"If the question doesn't appear to be answerable from the context provided, but seems to be a question about TypeScript, Javascript, or REST APIs, you may answer from outside of the provided context.",
		"Your name is DanGPT"
	],
	"searchType": "vector",
	"vectorSearch": {
		"column": "embeddings",
		"contentColumn": "content",
		"filter": {
			"section": "guide"
		}
	}
}
```

As you can see, the keyword search version has more settings, configuring fuzziness and boosters and column weights. The vector search only uses a filter. I would call this a plus for keyword search: you have more dials to tune the search and therefore get better answers. But it’s also more work, and the results from vector search are quite good without this tuning. 

In our case, we already have tuned the keyword search for our, well, docs search functionality. So it wasn’t necessarily extra work, and while playing with ChatGPT we discovered improvements to our docs and search as well. Also, Xata just happens to have a very nice UI for tuning your keyword search, so the work wasn’t hard to begin with (planning a separate blog post about that).

There is no reason for which vector search couldn’t also have boosters and column weights and the like, but we don’t have it yet in Xata and I don’t know of any other solution that makes that as easy as we make keyword search tuning. And, in general, there is more prior art to keyword search, but it is quite possible that vector search will catch up.

For now, I’m going to call keyword search a winner on this one.

Score: 3-4

## Convenience

Our documentation already had a search function, dog-fooding Xata, so that was quite simple to extend to a chat bot. Xata now also supports vector search natively, but using it required adding embeddings for all the documentation pages and figuring out a good chunking strategy. We have used the OpenAI embeddings API for producing the text embeddings, which had a minimal cost. Winner: Keyword search 

Score 3-5

## Latency

The keyword search approach needs an extra round-trip to the ChatGPT API. This adds in terms of latency to the result started to be streamed in the UI. By my measurements, this adds around 1.8s extra time 

**With vector search:**

![Xata bot request timing with vector search](/images/blog/keyword_or_vector/request_timing_vector.png)

**With keyword search:**

![Xata bot request timing with keyword search](/images/blog/keyword_or_vector/request_timing_keyword.png)

Note: The total and the content download times here are not relevant, because they mostly depend on how long the generated response is. Look at the “Waiting for server response” bar (the green one) to compare.

Winner: Vector search

Score: 4-5

## Cost

The keyword search version needs to do an extra API call to the ChatGPT API, on the other hand, the vector search version needs to produce embeddings for all the documents in the database plus the question. Unless we’re talking about a lot of documents, I’m going to call this a tie.

Score: 5-6

## Conclusion

The score is tight! In our case we have gone with using the keyword search for now, mostly because we have more ways of tuning it and as a result of that it generates slightly better answers for our set of test questions. Also, any improvements that we make to search automatically benefit both the search and the chat use cases. As we’re improving our vector search capabilities with more tuning options, we might switch to vector search, or a hybrid approach, in the future.

If you’d like to set up a similar chat bot for your own documentation, or any kind of knowledge base, you can easily implement the above using the Xata ask endpoint. [Create an account](https://app.xata.io/signup) for free and join us on [Discord](https://xata.io/discord). I’d be happy to personally help you get it up and running!

The blog post compares keyword search with semantic/vector search for the task of selecting context for an ChatGPT-based questions and answers bot.

Semantic or keyword search for finding ChatGPT context. Who searched it better?

Xata’s here to take care of the hard stuff surrounding data: high availability, scaling, access control, synchronization, and a bunch more. Our data platform is built to handle these problems, so you can focus on building what’s unique to your product. As developers ourselves, building a platform is fun. We use Xata in our own work as much as possible, and out of it comes a tonne of new ideas. These ideas, turned into features, help you move faster.

It's been hard to miss the thunderstorm generated by OpenAI. For the last few months, it’s been everywhere; simple answers to tricky questions, hip-hop songs about Kubernetes, or standup comedy about functional programming. From all the uses, one thing felt clear: this was a firm shove into the future. 

With their APIs, we now have tools to help you analyze, reshape, and reframe text. Tools to help inspire new ideas, distill information down into ELI5-chunks, or generate photo-realistic pictures of cats in space. It provides an API that deals with a complicated world. One where you can’t easily prepare for all reasonable inputs. 

This isn’t a teary-eyed post about OpenAI though. We’re wanting to show you something we’ve had brewing over the last few weeks. Let’s begin with a screenshot taken a couple of days ago, from ChatGPT.

![ChatGPT answering a question about Xata wrongly](/images/blog/chatgpt/chatgpt_xata_question.png)

Did you notice anything weird? 

For those familiar with Xata, you'll see that this not correct. ChatGPT has confidently given us a wrong answer. We can’t blame it though, the model has a documented cutoff date for its data in 2021 and unfortunately for us, Xata’s documentation was published after this date.

Let’s take a look at once more image. I’ll ask the same question again.

![DanGPT answering the question correctly](/images/blog/chatgpt/get_by_id_xata.png)

How’s that look to you? Hopefully familiar?

We’ve been working on integrating OpenAI into our data platform. Xata now lets you ask *your* data questions. It relies fully on the data you have inside Xata. No waiting for OpenAI, Google, or any other company to index your changes. It’s your data, and you’re fully in control of it. You choose when to add information, remove information, modify information. Xata takes these changes and makes them available nearly instantly. It requires no work from you, and comes with options for both free and pro users.

The idea came out of a [demo day](https://twitter.com/xata/status/1626631119306096658?s=61&t=kJ96Ir__sVvypRcu-eaOPw). We run a session every few weeks where the team gathers to show off ideas, proof-of-concepts, or features we’ve just completed. During the demo for the integration (in those days it was called DanGPT), I had my screen shared. Demos are bad at the best of times, and here, I wasn’t totally sure what would be returned.  I wrote my question, and nervously pressed a green “Ask” button that I had only finished that morning. The result that came out was almost perfect. It needed tuning, but seeing it take a natural-sounding question and coming back with Xata-specific details felt like magic. 

Building this feature has been the source of a buffet of new ideas for us. Here’s a few things we’re thinking about.

## Improve your documentation search

We use Xata to store our documentation. This is stored as markdown in `text` columns. The Xata Ask API queries the data and returns a response that matches the context of the question and uses the content we have stored. This helps users find information quicker, make use of it faster, and to be able to ask questions in a way that feels natural to them, rather than needing to try and adapt their question to what they think our docs wants.

This is coming to our documentation, look out for the “Ask our chat bot” buttons on the documentation site, and in the Xata app.

## Help your team get the information they need faster

If you have a lot of data, finding the right document can be tough. Finding the right text in that document can also be tough. A question might be phrased in a way that is uncommon, or an answer might require context from several documents.

Xata understands the intent of the question, and has to one column of one table all the way up to all columns in all tables. This means that it can take a wide range of questions, and stitch together a coherent answer.

## Augment customer support

Have a FAQ or receive common questions? Use Xata to help answer common questions before being escalated to an agent. The responses will adapt to the questions, and you can write documentation in natural language. Xata’s search service will make the data you change available almost immediately.

What’s got us excited about releasing our integration with OpenAI is that we can offer it on top of our existing platform. By using one of our SDKs or the REST API, you’re able to leverage this technology without any major changes to your workflow.

Here’s the API call our docs make to retrieve data from our docs table.

```jsx
POST https://xata-uq2d57.eu-west-1.xata.sh/db/docs:main/tables/search/ask
Accept: text/event-stream

{
    "question": "How do a retrieve a single record?",
    "rules": [
      "Do not answer questions about pricing or the free tier. Respond that Xata has several options available, please check https://xata.io/pricing for more information.",
            "When you give an example, this example must exist exactly in the context given.",
      "Only answer questions that are relating to the defined context or are general technical questions. If asked about a question outside of the context, you can respond with \"It doesn't look like I have enough information to answer that. Check the documentation or contact support.\"",
      "Your name is DanGPT"
    ],
    "searchType": "keyword",
    "search": {
      "fuzziness": 2,
      "prefix": "phrase",
      "target": [
        "slug",
        { "column": "title", "weight": 4 },
        "content",
        "section",
        { "column": "keywords", "weight": 4 }
      ],
      "boosters": [
        {
          "valueBooster": {
            "column": "section",
            "value": "guide",
            "factor": 18
          }
        }
      ]
    }
  }
```

[Sign up](https://app.xata.io/signup) to add ChatGPT to your data today and get started with this new [sample app](https://github.com/xataio/examples/tree/main/apps/sample-chatgpt). Come say hi on  our [Discord](https://xata.io/discord) if you have any questions.

We’re excited to see what you build 🦋

We’ve been working on integrating OpenAI into our data platform. Xata now lets you ask your data questions. It relies fully on the data you have inside Xata. No waiting for OpenAI, Google, or any other company to index your changes. It’s your data, and you’re fully in control of it. You choose when to add information, remove information, modify information. Xata takes these changes and makes them available nearly instantly.

Xata plus OpenAI: ChatGPT on your data

One of my favorite features of Xata is the built-in "fuzzy search"
functionality. Most database solutions let you search for an exact match. Users
these days, though, often expect a more forgiving search, one that will match
"olso" to "oslo" and "alez" to "alex".

Fuzzy search to the rescue! 💪

![The difference in results with fuzzy search enabled and disabled](/images/blog/fuzzy_search/xata-fuzzy-search-on-off.gif)

> **FYI:** The example code uses a Xata Worker using the Xata SDK + a React
> component using ReactQuery. However, you may use any type of server-side +
> client-side setup you are comfortable with.

## Fuzzy Search Is Enabled by Default

Xata search functionality comes with the `fuzziness` param set to `1` by
default, letting the user make one typo, such as one wrong character
("ale**z**") or one set of swapped characters ("o**ls**o") etc. It's a great
default, and the one we use for
[pruneyourfollows.com](https://pruneyourfollows.com).

Nonetheless I do like to explicitly state the fuzziness level in the Code,
together with the configuration for partially matching words.

```jsx
import React, {useState} from "react";
import {useDebounce} from "usehooks-ts";
import {useQuery} from "@tanstack/react-query";
import {xataWorker} from "./xata";

// Code executed on the server as a Cloudflare Worker
const searchAccount = xataWorker("searchAccount", async ({xata}, {term}) => {
  const results = await xata.search.all(term, {
    tables: [
      {
        table: "accounts",
        target: ["name", "username"],
      },
    ],
    // Fuzziness level
    fuzziness: 1,
    // Partially matching words
    prefix: "phrase",
  });

  return results;
});

// Code executed in the browser
export default function App() {
  const [term, setTerm] = useState("");
  const debouncedTerm = useDebounce(term, 300);

  const {data: results} = useQuery({
    queryKey: ["search", debouncedTerm],
    queryFn: () => {
      return searchAccount({term: debouncedTerm});
    },
    enabled: Boolean(debouncedTerm),
    placeholderData: [],
  });

  return (
    <main>
      <form onSubmit={(event) => event.preventDefault()}>
        <label htmlFor="search-field">Search accounts: </label>
        <input
          id="search-field"
          type="search"
          value={term}
          placeholder="Search accounts: name or username"
          onChange={(event) => {
            setTerm(event.target.value);
          }}
        />
      </form>

      <ul>
        {results.map(({record}) => {
          return (
            <li key={record.username}>
              <a href={`http://twitter.com/${record.username}`}>
                <img src={record.meta.profile_image_url} aria-hidden={true} />
                <span
                  dangerouslySetInnerHTML={{
                    __html: record.name,
                  }}
                />
                <br />
                @
                <span
                  dangerouslySetInnerHTML={{
                    __html: record.username,
                  }}
                />
              </a>
            </li>
          );
        })}
      </ul>
    </main>
  );
}
```

Other possible levels of `fuzziness` are `0`, to disable it altogether, and `2`
to extend the allowed typo tolerance. Higher than `2` is not supported.

## Highlight the Relevant Text

In addition to expecting a more forgiving search, users expect to know precisely
_why_ a result matches their search term. With a fuzzy search, that is even more
crucial, as it can lead to some unexpected matches.

When your mind thinks "oslo" but you wrote "olso", the "Espen Olson" results
might feel strange when not highlighted.

![The difference between results with and withuot highlighting](/images/blog/fuzzy_search/xata-highlights-on-off.gif)

Luckily, Xata generates ready-to-use HTML, wrapping the relevant matching text
in `<em>` for us. However, you need to use `getMetadata` on each record to get
access to it.

```jsx
import React, {useState} from "react";
import {useDebounce} from "usehooks-ts";
import {useQuery} from "@tanstack/react-query";
import {xataWorker} from "./xata";

// Code executed on the server as a Cloudflare Worker
const searchAccount = xataWorker("searchAccount", async ({xata}, {term}) => {
  const results = await xata.search.all(term, {
    tables: [
      {
        table: "accounts",
        target: ["name", "username"],
      },
    ],
    fuzziness: 1,
    prefix: "phrase",
  });

  // Getting the highlights (and more)
  const enrichedResults = results.map((result) => {
    return {
      ...result,
      ...result.record.getMetadata(),
    };
  });

  return enrichedResults;
});

//Code executed in the browser
export default function App() {
  const [term, setTerm] = useState("");
  const debouncedTerm = useDebounce(term, 300);

  const {data: results} = useQuery({
    queryKey: ["search", debouncedTerm],
    queryFn: () => {
      return searchAccount({term: debouncedTerm});
    },
    enabled: Boolean(debouncedTerm),
    placeholderData: [],
  });

  return (
    <main>
      <form onSubmit={(event) => event.preventDefault()}>
        <label htmlFor="search-field">Search accounts: </label>
        <input
          id="search-field"
          type="search"
          value={term}
          placeholder="Search accounts: name or username"
          onChange={(event) => {
            setTerm(event.target.value);
          }}
        />
      </form>

      <ul>
        {results.map(({record, highlight}) => {
          return (
            <li key={record.username}>
              <a href={`http://twitter.com/${record.username}`}>
                <img src={record.meta.profile_image_url} aria-hidden={true} />
                <span
                  dangerouslySetInnerHTML={{
                    __html: highlight.name || record.name,
                  }}
                />
                <br />
                @
                <span
                  dangerouslySetInnerHTML={{
                    __html: highlight.username || record.username,
                  }}
                />
              </a>
            </li>
          );
        })}
      </ul>
    </main>
  );
}
```

## Search Playground

Xata even allows you to play around with search—no code needed—using the Search
Engine Playground.

**Pro tip:** Use the "Get Code Snippet" feature to get a head start while
coding.

![Xata Playground showing similar code as this article](/images/blog/fuzzy_search/xata-search-engine-playground.png)

## Where To Go From Here?

I hope you are excited to dig into fuzzy search with Xata:

- The full [Prune your follows](https://pruneyourfollows.com/) code (Gatsby +
  Xata + Tailwind + ReactQuery) can be [found on Gitbub](https://github.com/queen-raae/prune-your-follows).
- The simplified demo (React + Vite + Xata + Tailwind + ReactQuery) used in this
  article can also be
  [found on Github](https://github.com/queen-raae/xata-fuzzy-search-demo).
- The Xata Docs
  [Full-text search article](https://xata.io/docs/typescript-client/search) has
  more details.

If you need any help implementing fuzzy search in your project, join the
[Xata Discord](https://xata.io/discord).

Xata is a serverless database built on PostgreSQL with data automatically replicated to Elasticsearch. This opens a whole new world of possibilities when it comes to free text search. Deep dive into Queen Raae's favorite Xata feature, fuzzy search.

How to Fuzzy Search with Xata

We’re always excited to learn how our users are using Xata and the value it provides to their business. Today we’ll be putting the community spotlight on [RankOne](https://www.rankone.global/), a social platform for streamers.

![RankOne profile.png](/images/blog/rankone/rankone.png)

RankOne is a startup focused on building the best social platform for gaming streamers. Streamers use RankOne to create personal gaming profiles and communicate their gaming preferences directly to their audience through a Twitch extension.

The RankOne platform is used by thousands of gamers and content creators. The team’s goal is to illuminate the gaming industry and bridge the gap between that industry and its gamers. In the past year, they’ve expanded into influencer marketing on Twitch and YouTube. Their unique knowledge of how people play games allows them to tailor marketing campaigns to reach the right audience. Piecing all these different datasets together, is where Xata comes in.

> The true serverless nature, the generous pricing plan and the developer experience was key for us when we picked Xata. <br />
> *Peter Spegel - Developer @ RankOne*

The gaming industry is booming, with over 5 million video games in existence and thousands of new titles being introduced each year. One of the benefits RankOne provides their users is being able to bring them closer to the games.

The team came across Xata as they were looking for a way to store information about all released and upcoming games in an easily searchable database. The fact that their application was built on a mix of TypeScript, Vite, Next.js, Node.js and React made Xata a perfect fit.

Everything RankOne does, from social profiles to personalized marketing campaigns, rely on game information and it’s vital to be able to access and query the data at all times.

> As a small startup we don't have the resources to maintain a large set of technologies but with Xata it was so easy to set everything up. <br />
> *Peter Spegel - Developer @ RankOne*

For the longest time, the platform was highly dependent on a costly third party API for game related data. With Xata they were able to move off that solution they’ve relied on for years in just a matter of days — saving money along the way and prepared for future scale and growth.

> We were blown away by the workspace environment, the documentation and the incredible response time on Discord. It's rare that you find a product that just works.  <br />
> *Peter Spegel - Developer @ RankOne*

If you’re a gamer, content creator or just generally interested in the use case check out [RankOne](https://www.rankone.global/) and follow them on [Twitter](https://twitter.com/rankoneglobal). And if you’d like to build your next big idea on Xata, [sign up](https://app.xata.io/signin) and come join our community on [Discord](https://xata.io/discord). We’re looking forward to seeing what you build next.

Interested in being featured on our community spotlight? [Send us an email](mailto:communityspotlight@xata.io) if you’d like to share your use case and use of Xata with our community.

How RankOne is using Xata to bring streamers closer to their audience.

How Xata powers search for RankOne’s social platform

Lee Robinson, of Vercel, recently wrote a blog post summarising the [2023 State of Databases for Serverless & Edge](https://leerob.io/blog/backend) . It’s a great overview that shines a light on what’s new on the horizon. In the trends section, listed first, we find:

> Databases are increasingly becoming **data platforms**, including other adjacent solutions like full-text search and analytics.

Yes! That sounds a lot like Xata, we are presenting ourselves as a [Serverless Data Platform](https://xata.io/docs/concepts/serverless-data-platform) and we have full-text search and analytics built-in on top of the normal transactional/relational database. 

I would define a data platform as a service that offers multiple types of data infrastructure integrated in a single Cloud service. For example, Xata is offering a database service on top of the PostgreSQL + Elasticsearch/OpenSearch combination, and we integrate them deeply into a single service. We automatically replicate the data between the two, and have a common consistent API, plus offer the same developer experience (including branches), and common SDKs. 

## Examples

The trend is larger than just Xata, and there are a number of “database startups” that combine functionality from multiple types of stores or data infra in order to offer more convenience to their users. 

Let’s look at some examples (alphabetically sorted):

- [AlloyDB](https://cloud.google.com/alloydb), from Google, combines OLTP with OLAP, and uses the PostgreSQL wire protocol as an interface.
- [Grafbase](https://grafbase.com/) is using DynamoDB internally and they plan to add free-text-search.
- [MongoDB Atlas](https://www.mongodb.com/atlas/database) is obviously using MongoDB but it is also offering free-text-search based on Lucene.
- [PlanetScale](https://planetscale.com/) is presenting themselves as a MySQL platform. In addition to MySQL, it uses Vitess for sharding and things like migrations, and they also have added caching infrastructure to their service.
- [Supabase](https://supabase.com/) is based on PostgreSQL but also offers blob storage and image storage services.
- [Tigris Data](https://www.tigrisdata.com/) is based on FoundationDB and are replicating the data into Meilisearch for free-text-search.
- [TimeScale](https://www.timescale.com/) adds timeseries functionality to PostgreSQL.
- [Upstash](https://upstash.com/) offers serverless Redis and Kafka with HTTP APIs for both.

There are probably more, if you have other examples or corrections to the above, [let me know](https://twitter.com/tudor_g) and I will incorporate them.

As you can see, on top of the core database functionality, it’s becoming more and more common to see things like caching / edge caching infrastructure, search engines, blob storage, OLAP functionality, and time-series support.

![Data platform meme](/images/blog/platforms/captain-data-platform.jpg)

## Benefits of Data Platforms

To understand why this trend is happening, let’s look at what business value data platforms offer their customers.

### Less Glue Code

If, for instance, data is automatically replicated between the database and the search engine, or the cache is invalidated when the data changes, this reduces the amount of code you need to write and maintain. This type of code is often difficult to get right and not particularly rewarding, so most teams would be grateful if someone else handled it.

### Less Ops

We’re seeing many companies using many (>5) data stores and to operate them means they have to understand each of them pretty deeply: when to use each one, how to configure them, how to monitor them, how to optimize them, and so on. Even if they use managed services that take some of the operational load, the inconsistencies and the need to move the data between them ultimately means a lot of work, and teams are struggling with this.

### Better DX

Having consistent APIs, common SDKs, and — not less important — consistent documentation across data products simplifies the job of the developer. For example, the concept of branches has recently expanded to databases for deployment previews and for managing schema changes, but you only really reap the benefits of them if they work for all your data stores.

### Faster Development

Want to add relevancy based free-text-search to your data table? You just call another API endpoint. There’s no new infrastructure to set up, and no replication of data. The data is already indexed for search and it’s ready for you to use it. Want to add edge caching for your queries? Just add a parameter. Smart cache invalidation is done automatically.

These examples show how integrated data platforms can make developing features a lot faster.

### Empowers More Developers

The data platforms have an opportunity to offer new abstractions on top of existing database technology. These abstractions can be better fit for modern tech stacks. In particular, frontend engineers that normally don’t get excited about databases are now enabled to create more complex and scalable applications than ever before. Building on a data platform can feel as simple as building on top of a headless CMS.

## Integration and DX are the keys

A key aspect here is how good the integration is. AWS already offers database services of all kinds and forms but they are tragically siloed. They have different APIs, different SDKs, different quality of documentation, different ways to configure and monitor them. This makes for clunky developer experience.

Modern data platforms can differentiate themselves at the infra layer and at the DX layer.

![Data platform layers](/images/blog/platforms/data-platform.png)

At the infra layer, data platforms can simplify the developers life by offering functionality that normally requires custom code. A few examples:

- keep data in sync between the DB and the search engine
- automatic cache invalidation when the data in the DB changes
- automatically delete files in the blob storage when the associated DB rows are deleted
- replicate data across regions globally

At the DX layer, data platforms can offer:

- API, SDK, and docs consistency
- better developer workflow via branches, safe migrations, and deploy previews
- less required configuration

## The not-so-distant future

Will this trend continue to grow? Time will tell, but we think it will.

If you’ve read this far, you might want to check out [Xata](http://xata.io), the scalable serverless database with built-in search that supports branching and works well with your existing TypeScript and Github workflows.

Lee Robinson, of Vercel, recently wrote a blog post summarising the 2023 State of Databases for Serverless & Edge. In the trends section, listed first, we find: "Databases are increasingly becoming data platforms, including other adjacent solutions like full-text search and analytics." This blog post analysis this trend.

A New Trend for Databases: Platforms

Building an application is not easy, or to be more accurate, this usually starts easy and becomes complicated. Let’s see why. What are the usual challenges we need to deal with?

## User Management

Every application has users, and this comes with a lot of specific problems:

- How do my users log in?
- How do I store their password? This is quite sensitive.
- What if I want to connect through my Google or GitHub account?
- But, if a user did connect with an email and after with Github. Should I link the accounts?
- How can I ban a user?
- …

Luckily, [Clerk](https://clerk.dev/) solves all those issues for us (with a nice free tier). The only thing we need to care about is storing the `userId` and making sure a user can only see his own data.

## Features Management

One of the main challenges in building an application is to add or remove features. Indeed, without the correct tooling, it’s easy to forget something. Leading to a bug and unhappy customers at the end (or even worse if you are discovering a critical bug on a Sunday).

Again, we can help ourselves using the correct tool or, more precisely, the combination of tools!

Using TypeScript as a base, in combination with Xata and [tRPC](http://trpc.io), we can have full type safety from the database schema to our React components.

For example, if you decide to remove a field from your database, with this setup in place, you can run `xata codegen` and use TypeScript to track **every** usage of this field in your **entire** application, backend side & frontend side.

Of course, this is not 100% bullet-proof guarantee. But trust us: this will keep you on the safe side for most cases.

## Getting Started!

To help you in this journey, Xata has a starter with already everything set up for you.

The starter has a simple multi-user Todo application, this should provide you with enough guidance to build a solid stack for your next amazing project!

To get started, let’s start a new project together and see how everything works.

```bash
$ npx degit xataio/examples/apps/starter-nextjs-trpc-clerk my-todo-clerk-app
$ cd my-todo-clerk-app
$ npm install # yarn install or pnpm install
```

So far so good, now let’s connect our application to Clerk and Xata.

Go [Xata](https://xata.io) and create an account. Login and connect the project:

```bash
$ npm run xata -- auth login # needed only once$ npm run xata:init
```

Now create an account and a project at [Clerk](https://clerk.dev).

You can then go to [Clerk + Next.js integration documentation](https://clerk.dev/docs/nextjs/set-environment-keys) and copy the credentials to your `.env` file. That’s it! Just run `npm run dev` and you have a working application.

## Code Walkthrough

```tsx
// server/trpc.ts
import { initTRPC, TRPCError } from "@trpc/server";
import { Context } from "./context";

const t = initTRPC.context<Context>().create();

const isAuthed = t.middleware(({ next, ctx }) => {
  if (ctx.user === null) {
    throw new TRPCError({ code: "UNAUTHORIZED" });
  }

  return next({
    ctx: {
      user: ctx.user,
    },
  });
});

export const router = t.router;
export const publicProcedure = t.procedure;
export const protectedProcedure = t.procedure.use(isAuthed);
```

This is where everything starts, we are exposing two procedures, one for public routes and one with auth. All the clerk logic lives in `server/context.ts`, and the context is injected `pages/api/trpc/[trpc].ts`.

The idea is quite simple, if Clerk has some information about the user, it’s injected in the `context`, checking if the user is here, we can’t filter authenticate users. Let’s take `todo/list` as an example:

```tsx
// server/routers/todo.ts
import { protectedProcedure, router } from "../trpc";
import { z } from "zod";
import { getXataClient } from "../../utils/xata";
import { TRPCError } from "@trpc/server";

export const todoRouter = router({
  /**
   * List user todo items.
   */
  list: protectedProcedure.query(({ ctx }) => {
    const userId = ctx.user.id; // here `ctx.user` is typed as `User`
    const xata = getXataClient();

    return xata.db.items.filter({ userId }).getAll();
  }),
  /* ... */
});
```

Here are using `protectedProcedure` since we need the user information for our database query. The entire function is type-safe, I have access to all user’s data in this function and we are simply using the `user.id` as a filter. Please note that we don’t have any types annotation as so ever and just returning raw Xata response.

Looking at `pages/index.ts`, we can look at how to use this route in our application.

![https://user-images.githubusercontent.com/1761469/209673800-d4915411-53a4-4b08-9553-31e5e592496d.png](https://user-images.githubusercontent.com/1761469/209673800-d4915411-53a4-4b08-9553-31e5e592496d.png)

Here, everything is type-safe, serialized through the tRPC layer, and ready to be consumed.

On the React part, tRPC provides you a typed version of react-query, this means you will have all the benefits of React-Query like the query cache but without having to define your own fetcher.

```tsx
// Function block of `Hello` inside `pages/index.tsx`

const utils = trpc.useContext();

const { mutate: addTodo } = trpc.todo.add.useMutation({
  onMutate: async (newTodo) => {
    await utils.todo.list.cancel();

    const previousTodos = utils.todo.list.getData();

    const optimisticTodo: RouterOutputs["todo"]["list"][-1] = {
      title: newTodo.title,
      createdAt: new Date().toISOString(),
      userId: "me",
      id: "optimistic_" + self.crypto.randomUUID(),
      isCompleted: false,
    };
    utils.todo.list.setData(undefined, (old) =>
      old ? [...old, optimisticTodo] : [optimisticTodo]
    );

    return { previousTodos };
  },
  onError: (_err, _newTodo, context) => {
    utils.todo.list.setData(undefined, context?.previousTodos);
  },
  onSettled: () => {
    utils.todo.list.invalidate();
  },
});
```

A lot of interesting patterns in this `addTodo` mutation! First, we are using `trpc.useContext()`, this is the way to access react-query client with all the types backed! Here, we are implementing a simple optimistic UI: we don’t wait for the backend response to update the UI state!

To do so, we need to create a fake entry in our cache, with the same type as our `todo/list` return type. We can simply use this `RouterOutputs["todo"]["list"][-1]` that will infer the type from our router.

Doing so, if tomorrow somebody adds a property to this todo list, this part of the code will not compile until we provide a value for the introduced property, one less bug to fix later!

You also have `RouterInputs` and `ReactQueryOptions` as inferred types exported inside `utils/trpc.ts`. Those helpers are useful to avoid duplicating types. As code, we should always try to avoid duplicate types when we can, so just changing the return type or the parameter of one of our routes will propagate through the entire codebase.

## Final Considerations

And this is it for the big lines, the easiest way to learn more about the project is to try to destroy it! Add or remove some code and see how everything plays together.

Fork this [example](https://github.com/xataio/examples/tree/main/apps/starter-nextjs-trpc-clerk) and sign up with [Xata](https://xata.io) today to get started!

Have fun! If you have any suggestions on how to improve this starter or encounter any friction, we are available on [our Discord channel](https://xata.io/discord)


Let’s build a fully typed application, using Clerk as an authentication layer and tRPC to help us having a robust and easy to maintain stack.

Xata, tRPC, and Clerk: The Killer Combo!

Late last year, we launched the [Xata Challenge](https://twitter.com/xata/status/1600186056942489600) to challenge our community to build that app that’s been itching at them all year. We were so excited to see the engagement from our community and all of the amazing submissions. The deadlines for submissions have now passed and we’re thrilled to announce the winners of the first ever Xata Challenge!

🥁 *Drum roll, please* 🥁 

And the three winners, as voted by the [Xata team](https://xata.io/about), are…

## ishn.xyz Link Shortener

![ishn.xyz Link Shortener](/images/blog/xata-challenge-winners/ishn-link-shortener.png)

Builder: [Ishaan Bedi](https://twitter.com/ishnbedi)

Repository: [github.com/ishaanbedi/ishn.xyz](https://github.com/ishaanbedi/ishn.xyz)

URL: [ishn.xyz](https://www.ishn.xyz/)

ishn.xyz is a fast, feature-rich and completely free link shortener. It has features like custom slugs, statistical tracking, QR Code generation and instant sharing. The Xata team really liked how simple and easy to use this solution was, and that an API was also part of the offering itself. We're really excited to see where Ishaan takes this product suite next.

## Thoughtnaut

![Thoughtnaut](/images/blog/xata-challenge-winners/thoughtnaut.png)

Builder: [Justin Juno](https://twitter.com/justinjunodev)

Repository: [github.com/justinjunodev/thoughtnaut.io](https://github.com/justinjunodev/thoughtnaut.io)

URL: [thoughtnaut.io](https://www.thoughtnaut.io/)

Thoughtnaut is a journaling companion that will explore your mind and help you find clarity. The Xata team found Justin's app to be incredibly unique and makes it easy to take a step back and self reflect. Thoughtnaut is already a favorite amongst Xataflies and is fueling journals across the team.

## Apprendamos

![Apprendamos](/images/blog/xata-challenge-winners/apprendamos.png)

Builder: [Anthony Cueva](https://twitter.com/cuevatnt)

Repository: [github.com/apprendamos/apprendamosdotcom](https://github.com/apprendamos/apprendamosdotcom)

URL: [apprendamos.com](https://apprendamos.com/home)

Apprendamos is a social networking platform where people can share articles and interact with others on the network. We were excited to see the first social networking platform built on Xata and the complexity involved with the implementation itself. Not to mention, It's use case we're currently handling in our #random slack channel. It'll be nice to have something a bit more tailored to our article sharing moving forward.

Congratulations to all our winners who will receive a $500 cash prize and our participants who will be receiving some one of a kind Xata swag! 🍾 🎉 🦋

If you missed out on the Xata challenge this time around, don't worry. We'll be hosting another one soon. In the meantime if you'd like to showcase what you've built on Xata, come share with us on [Discord](https://xata.io/discord).

We're excited to announce the winners of the first Xata Challenge. There were many great submissions, here are the top three as voted by the Xata team.

Announcing the winners of the Xata Challenge!

![Cover image](https://blog.cloudflare.com/content/images/2022/11/Customer-Story_-Xata_light.png)

In the last few years, there's been a rise of Jamstack, and new ways of thinking about the cloud that some people call serverless or edge computing. Instead of maintaining dedicated servers to run a single service, these architectures split applications in smaller services or functions.

By simplifying the state and context of our applications, we can benefit from external providers deploying these functions in dozens of servers across the globe. This architecture benefits the developer and user experience alike. Developers don't have to manage servers, and users don't have to experience latency. Your application simply scales, even if you receive hundreds of thousands of unexpected visitors.

When it comes to databases though, we still struggle with the complexity of orchestrating replication, failover and high availability. Traditional databases are difficult to scale horizontally and usually require a lot of infrastructure maintenance or learning complex database optimization concepts.

At Xata we are building a modern data platform designed for scalable applications. It allows you to focus on your application logic, instead of having to worry about how the data is stored.

## Making databases accessible to everyone

We started Xata with the mission of helping developers build their applications and forget about maintaining the underlying infrastructure.

With that mission in mind, we asked ourselves: how can we make databases accessible to everyone? How can we provide a delightful experience for a frontend engineer or designer using a database?

To begin with, we built an appealing web dashboard, that any developer --- no matter their experience --- can be comfortable using to work with their data.

Whether they're defining or refining their schema, viewing or adding records, fine-tuning search results with query boosters, or getting code snippets to speed up development with our SDK. We believe that only the best user experience can provide the best development experience.

We identified a major challenge amongst our user base early on. Many front-end developers want to access their database from client-side code.

Allowing access to a database from client-side code comes with several security risks if not done properly. For example, someone could inspect the code, find the credentials, and if they weren't scoped to a single operation, potentially query or modify other parts of the database. Unfortunately, this is a common reason for data leaks and security breaches.

It was a hard problem to solve, and after plenty of brainstorming, we agreed on two potential ways forward: implementing row-level access rules or writing API routes that talked to the database from server code.

Row-level access rules are a good way to solve this, but they would have required us to define our own opinionated language. For our users, it would have been hard to migrate away when they outgrow this solution. Instead, we preferred to focus on making serverless functions easier for our users.

Typically, serverless functions require you to either choose a full stack framework or manually write, compile, deploy and use them. This generally adds a lot of cognitive overhead even to choose the right solution. We wanted to simplify accessing the database from the frontend without sacrificing flexibility for developers. This is why we decided to build Xata Workers.

## Xata Workers

A Xata Worker is a function that a developer can write in JavaScript or TypeScript as client-side code. Rather than being executed client-side, it will actually be executed on Cloudflare's global network.

You can think of a Xata Worker as a `getServerSideProps` function in Next.js or a `loader` function in Remix. You write your server logic in a function and our tooling takes care of deploying and running it server-side for you (yes, it's that easy).

The main difference with other alternatives is that Xata Workers are, by design, agnostic to a framework or hosting provider. You can use them to build any kind of application or website, and even upload it as static HTML in a service like GitHub Pages or S3. You don't need a full stack web framework to use Xata Workers.

With our command line tool, we handle the build and deployment process. When the function is invoked, the Xata Worker actually makes a request to a serverless function over the network.

```ts
import { useQuery } from '@tanstack/react-query';
import { xataWorker } from '~/xata';

const listProducts = xataWorker('listProducts', async ({ xata }) => {
  return await xata.db.products.sort('popularity', 'desc').getMany();
});

export const Home = () => {
  const { data = [] } = useQuery(['products'], listProducts);

  return (
    <Grid>
      {data.map((product) => (
        <ProductCard key={product.id} product={product} />
      ))}
    </Grid>
  );
};

```

In the code snippet above, you can see a React component that connects to an e-commerce database with products on sale. Inside the UI component, with a popular client-side data fetching library, data is retrieved from the serverless function and for each product it renders another component in a grid.

As you can see a Xata Worker is a function that wraps any user-defined code and receives an instance of our SDK as a parameter. This instance has access to the database and, given that the code doesn't run on the browser anymore, your secrets are not exposed for malicious usage.

When using a Xata Worker in TypeScript, our command line tool also generates custom types based on your schema. These types offer type-safety for your queries or mutations, and improve your developer experience by adding extra intellisense to your IDE.

![SDK Intellisense](https://blog.cloudflare.com/content/images/2022/11/Untitled.png)

A Xata Worker, like any other function, can receive additional parameters that pass application state, context or user tokens. The code you write in the function can either return an object that will be serialized over the network with a superset of JSON to support dates and other non-primitive data types, or a full response with a custom status code and headers.

Developers can write all their logic, including their own authentication and authorization. Unlike complex row level access control rules, you can easily express your logic without constraints and even unit test it with the rest of your code.

## How we use Cloudflare

We are happy to join the [Supercloud](https://blog.cloudflare.com/welcome-to-the-supercloud-and-developer-week-2022/) movement, Cloudflare has an excellent track record, and we are using[ Cloudflare Workers for Platforms](https://developers.cloudflare.com/cloudflare-for-platforms/workers-for-platforms/) to host our serverless functions. By using the Workers isolated execution contexts we reduce security risks of running untrusted code on our own while being close to our users, resulting in super low latency.

All of it, without having to deploy extra infrastructure to handle our user's application load or ask them to configure how the serverless functions should be deployed. It really feels like magic! Now, let's dive into the internals of how we use Cloudflare to run our Xata Workers.

For every workspace in Xata we create a Worker Namespace, a Workers for Platform concept to organize Workers and restrict the routing between them. We used Namespaces to group and encapsulate the different functions coming from all the databases built by a client or team.

When a user deploys a Xata Worker, we create a new Worker Script, and we upload the compiled code to its Namespace. Each script has a unique name with a compilation identifier so that the user can have multiple versions of the same function running at the same time.

During the compilation, we inject the database connection details and the database name. This way, the function can connect to the database without leaking secrets and restricting the scope of access to the database, all of it transparently for the developer.

When the client-side application runs a Xata Worker, it calls a Dispatcher function that processes the request and calls the correct Worker Script. The dispatcher function is also responsible for configuring the CORS headers and the log drain that can be used by the developer to debug their functions.

![A diagram featuring Xata worker workflow with Cloudflare](https://blog.cloudflare.com/content/images/2022/11/Untitled--1-.png)

By using Cloudflare, we are also able to benefit from other products in the Workers ecosystem. For example, we can provide an easy way to cache the results of a query in Cloudflare's global network. That way, we can serve the results for read-only queries directly from locations closer to the end users, without having to call the database again and again for every Worker invocation. For the developer, it's only a matter of adding a "cache" parameter to their query with the number of milliseconds they want to cache the results in a KV Namespace.

```ts
import { xataWorker } from '~/xata';

const listProducts = xataWorker('listProducts', async ({ xata }) => {
  return await xata.db.products.sort('popularity', 'desc').getMany({
    cache: 15 * 60 * 1000 // TTL
  });
});
```

In development mode, we provide a command to run the functions locally and test them before deploying them to production. This enables rapid development workflows with real-time filesystem change monitoring and hot reloading of the workers code. Internally, we use the latest version of miniflare to emulate the Cloudflare Workers runtime, mimicking the real production environment.

## Conclusion

Xata is now out of beta and available for everyone. We offer a generous free tier that allows you to build and deploy your applications and only pay to scale them when you actually need it.

You can [sign up for free](https://app.xata.io/), create a database in seconds and enjoy features such as branching with zero downtime migrations, search and analytics, transactions, and many others. Check out our [website](https://xata.io/) to learn more!

Xata Workers are currently in private beta. If you are interested in trying them out, you can sign up for the [waitlist](https://xata.io/beta/workers) and talk us through your use case. We are looking for developers that are willing to provide feedback and shape this new feature for our serverless data platform.

We are very proud of our collaboration with Cloudflare for this new feature. Processing data closer to where it's being requested is the future of computing and we are excited to be part of this movement. We look forward to seeing what you build with Xata Workers.

Xata Workers act as middleware to simplify database access and allow their developers to deploy functions that sit in front of their databases.

Xata Workers: client-side database access without client-side secrets

[Hackmamba](https://hackmamba.io/) is a great resource for creating technical content and strategy. We were excited to partner with them in their 2022 hackathon focused on developing participants technical writing skills.

Learning, writing, and networking during two full weeks! What more could you ask for? This last Hackmamba Jamstack Content Hackathon 2.0. was quite a journey.

The rules were simple: build whatever you want using Xata & Cloudinary and write a blog article about it.

This was a vibrant experience for us at Xata, since we also launched the product the same week as the hackathon. The Hackmamba community and participants were wonderful and built some great apps and content on top of Xata.

Here are the winners:

1st Place: Gift Uhiene https://dev.to/hackmamba/build-a-full-stack-jamstack-application-with-xata-cloudinary-and-nextjs-50pd

2nd place: Johnpaul Eze https://dev.to/hackmamba/modern-e-commerce-with-xata-and-cloudinary-foc

3rd place: Ubaydah Abdulwasiu https://dev.to/hackmamba/how-to-build-an-online-library-in-nextjs-with-xata-and-cloudinary-26b4

Congratulations to them and all the participants! 🎉 🎉

You can find all published posts https://dev.to/hackmamba

These articles are great if you're looking for some good examples to get started with Xata. If you're looking to try Xata out as the year winds down, be sure to enter your app in the [Xata Challenge](https://xata.io/challenge)!

Come chat with us on [discord](https://xata.io/discord) if you have any questions or simply want to talk data.

Learning, writing, and networking during two full weeks! This last Hackmamba Jamstack Content Hackathon 2.0. was quite a journey!

Hackmamba Jamstack Content Hackathon 2.0

Diversity is one of the most important aspects of Xata’s culture. You'll see it in everything we build. Which is why we continue to add new JavaScript frameworks to our [examples repo](https://github.com/xataio/examples). Here are a few more to help you along on your frontend journey.

Please keep in mind that whatever framework you pick, it needs to support server-side rendering and/or endpoints support. Indeed, any call to Xata needs to be server-side to avoid leaking credentials.

Let's take a quick tour of our new options for starters

## SolidStart

[SolidStart](https://start.solidjs.com/getting-started/what-is-solidstart) is the meta-framework on top of SolidJS and Solid router. You can use Xata in any server-side function. In the SolidStart world, you can use Xata inside `createServerData$` and `createServerAction$` functions.

To try out easily, you can use the SolidStart starter: `npx degit xataio/examples/apps/starter-solidstart solidstart-sample-app`

## Astro

[Astro](https://astro.build/) works a lot of JavaScript frameworks out-of-the-box as React, SolidJS or VueJS. Astro also has an SSR built-in feature, and this is precisely what we need to be able to query Xata securely!

This is the starter for Astro: `npx degit xataio/examples/apps/starter-astro astro-sample-app`

A few highlights, the server is set in SSR mode, and you will need a specific adapter depending on where you want to deploy application.

For example, if I want to deploy to netlify, I can run `npx astro add netlify`. Please note that if you don’t want to setup Astro in SSR mode, you can, but you will need to rebuild your application whenever you want new data. Indeed, without SSR mode enabled, the data are requested and injected at build time!

## SvelteKit

Last but not least, from the new starers list -- [SvelteKit](https://kit.svelte.dev/)! Another great option is to have SSR support with TypeSafety between your frontend and backend (and database if you are using Xata!).
Please note that you need to have the dev server running to generate some types (I figured this the hard way).
All your database calls need to be in `+page.server.ts` to stay server-side. You can load any data in `load` and react to any action in `actions` ; very straight forward!

As always, this is the command to let you play by yourself:
`npx degit xataio/examples/apps/starter-sveltekit sveltekit-sample-app`

## Final note

What ever flavor you are choosing, my advice (and this is just my point of view), is to check if you have a end to end type safety. This is important because when you will run `xata codegen`, TypeScript will compile or not, telling you what needs to be adjust in your application.

Therefore, you are more confident that your application doesn’t consume stale data, so no bugs, and happy customer at the end :)

If you encounter any issues or you just want to talk, please join us on [discord](https://xata.io/discord)

To help in your journey with Xata, we are providing more and more starters! Today will be about SolidStart, Astro, and SvelteKit. Have fun!

New starters!

Starting today, Xata is out of private Beta! 

Goodbye waiting list 👋, you can go ahead and [create an account](https://app.xata.io/signup). If you like to learn by doing and want to dive in right now, we promise you’re going to have a production-ready database, including high availability, with a data model, and test data in it in a matter of minutes. [Try it](https://app.xata.io/signup)!

After more than a year of work and receiving and addressing feedback from over 700 developers, we are very excited to show you what we’ve been up to. 

So let us take you on a quick tour.

## Why Xata?

Our mission is to take the hard data problems, solve them using the industry's best practices, and then offer them to you in the most easy-to-use way possible. 

The hard data problems include running PostgreSQL at scale and with high availability, replicating the data into Elasticsearch for search and analytics, edge caching, major-version upgrades, data migrations, observability, and so on. For more details around how we abstract away this complexity, see the How it Works documentation page.

![Xata technical diagram](/images/docs/xata-cell.png)

We call the type of service we’re building at Xata a [Serverless Data Platform](https://xata.io/docs/intro/serverless-data-platform), and we think it’s the next logical evolution after Database as a Service (DBaaS).  Similar to how platforms like Vercel, Netlify, or Cloudflare Pages leverage CDNs, serverless and edge functions, to host your app globally and give you a simple developer experience, Xata does the same for the data infrastructure.

When we say that we offer them to you in the most easy-to-use way possible, we mean things like:

- as serverless as it gets (see our blog post on [serverless databases](https://xata.io/blog/what-is-a-serverless-database))
- a single consistent API across multiple types of data stores (relational DB, search/analytics engine, in the future caching and queues)
- a web UI that makes working with the data easy during and after development
- ORM-like high-level clients for the most popular programming languages

## Show me the Code

Xata has a relational data model, with a strict schema (schemaful) and support for JSON-like objects. Records are grouped into tables, which are grouped into databases. Xata supports rich column types and relations between tables can be represented via link columns, which are similar to foreign keys.

A record in Xata can look something like this:

```json
{
  "name": "John Doe",
  "email": "john@example.com",
  "age": 42,
  "address": {
    "street": "123 Main St",
    "city": "New York"
  },
  "labels": ["admin", "user"],
	"team": "rec_cdcp6a98ru583kilecog"
}
```

The above shows a column of type object (`address`) with two sub-keys (`street` and `city`). It also shows a column of type email and an array-like column (`labels` of type `multiple`).

![Schema editor](/images/blog/launch/schema-editor.mp4)

The `team` column is a **link column**, pointing to a record in another table, similar to a foreign key. Link columns make it easy to represent relationships, see more in our [Data model](https://xata.io/docs/concepts/data-model) docs.

With the TypeScript/Javascript SDK, you can query the above record with something like this:

```tsx
const users = await xata.db.Users.select(["*", "team.*"])
    .filter({ "address.city": "New York" })
    .getMany();
```

In the above, because `team.*` was explicitly requested, the results will return all the columns from the linked `Teams` table as well. Behind the scene, Xata is doing a JOIN, but we expose it in a simpler way in our APIs.

You know what’s the best part about the above? If you’re using TypeScript, both the parameters and the results are fully typed. So you can easily use auto-complete to get column names, for example, and the types are going to catch mismatches between the code and the database schema. Type safety with no effort!

You can see a lot more examples of how to use both the API and the TypeScript SDK in our [API Guide](https://xata.io/docs/api-guide/get).

*Psst, there’s an easier way!*

You **can** read the API guide to see all the query options, but the truth is you don’t really have to 😀. Instead, you can configure your query in the UI and then hit the “Get code snippet” button. It will give you the exact query that was executed to get the results that you see on the screen. We believe that our code snippets are not only the fastest way to develop on top of Xata but also the best way to learn Xata.

![Get Code Snippet](/images/blog/launch/get-code-snippet.mp4)

## Branches and Zero-Downtime Migrations

A database in Xata can have multiple branches, which are first-class citizens in the API. Branches are full logical databases that might or might not exist on the same set of physical instances. When you start a new branch from an existing one, the schema is copied, but the data is not automatically available in the new branch. As you create a development branch from a production branch, you can copy and sanitize parts of the data or initialize the branch with test data.

For each branch, we record its schema history in a list of “commits”. These commits contain operations like “add a column”, “rename a column”, “add a new table”, “delete a table”, and so on. We also record which branch is created from which. This history information allows us to create the correct migration diffs and perform the minimum set of operations required for a migration.

There are multiple benefits to having database branches, which is why they are being adopted by multiple database products recently. They:

- enable development workflows that match git-workflows, for example having a branch for development, one for staging, one for production, etc.
- enable deployment previews on platforms like Vercel and Netlify.
- enable standardized workflows for applying schema migrations safely.

The last one is particularly interesting because migrating database schemas is usually easy with small amounts of data, but becomes very painful when dealing with large amounts of data. In the NoSQL wave of DB products, this issue was avoided by going schemaless. However, not having a schema introduces its own class of problems, so lately the pendulum is swinging back toward databases with strict schemas. 

Because we have control over the data model and because Xata has native concepts of migrations and migration requests, we can ensure that all supported migration types are safe and they don’t lock the database. That’s why we call our migrations **zero-downtime** migrations. They constrain you in what types of migrations exist, but you can have the confidence that your data model will keep working at scale.

## Free-Text Search

As we have shown earlier, we store the data in PostgreSQL, but we also replicate it to Elasticsearch. This enables us to offer significantly more advanced free-text search compared to the free-text-search support from PostgreSQL. This includes:

- [fuzinness control](https://xata.io/docs/api-guide/search#fuzziness-and-typo-tolerance) so that you can have typo tolerance in your search.
- [column weights](https://xata.io/docs/api-guide/search#column-weights), so you can value matches in some columns more. For example, you can boost matches in the “title” column more than matches in the rest of the columns.
- [numeric boosters](https://xata.io/docs/api-guide/search#numeric-booster), so you can boost records by a numeric field, for example by “rating”.
- [date boosters](https://xata.io/docs/api-guide/search#date-booster), so you can boost records that are more recent in time.
- [value boosters](https://xata.io/docs/api-guide/search#exact-value-booster), so you can boost records with a particular column value, for example, by “category”.

This allows you to build the perfect search experience for your users, without having to use a different service or product and worrying about how to replicate the data. 

Since relevancy tuning can be tricky business, we provide a UI so you can do it visually, see the results of it in real-time, then hit the Get Code Snippet to get the code that you need in your app.

![Search](/images/blog/launch/search.mp4)

### Summaries and Aggregations

Xata offers two ways to aggregate your data: summaries, which are executed in PostgreSQL, and aggregations, which are executed in Elasticsearch.

An example of a relatively complex visualization that can be created with the [Aggregation API](https://xata.io/docs/api-guide/aggregate) could be: a multi-line chart, where each line represents a movie genre, and the Y axis represents the average rating of the movies in that genre, per year. This chart can be obtained with a single aggregation request, looking something like this:

```tsx
const results = await xata.db.titles.aggregate({
  "movieGenres": {
    topValues: {
      column: "genre",
      size: 50
      aggs: {
        "byReleaseDate": {
          dateHistogram: {
            column: "releaseDate",
            calendarInterval: "year",
          },
          aggs: {
            "avgRating": {
              average: {
                column: "rating"
              }
            }
          }
        }
      }
    }
  },
})
```

You can find more examples in the [API guide](https://xata.io/docs/api-guide/aggregate). We’d don’t yet have a UI to help you build this type of aggregations, but if you know us by now, we’re going to have one soon.

### And we’re just getting started

With today’s public launch, it is day 0 at Xata. Our mission to simplify the way developers work with data gives us a strong focus on what features to add to the product. We believe that Xata, as a [serverless data platform](https://xata.io/docs/intro/serverless-data-platform), is the perfect companion for full-stack frameworks like Next.js, Remix.run, SvelteKit and others.

We’re very excited about upcoming features like Edge Functions and Caching (currently in private Beta, see more here),  integrations into Vercel and Netlify, global data replication, and many others. [Stay tuned](https://twitter.com/xata), there’s so much more to come.

### How free is free?

Many DBaaS and BaaS vendors offer a free-tier plan, but usually, it doesn’t include redundancy or high availability, and it’s often “for development” only with limitations based on activity. 

Our [free tier](https://xata.io/pricing) is explicitly recommended for production. We want you to build your personal projects, non-profit web apps, and fun apps on top of Xata, and you shouldn’t have to pay for it. You can also freely collaborate with others and be part of multiple teams (up to 40 workspaces per user). The limitations of the service are clearly stated on the [pricing page](https://xata.io/pricing) and on the [limits](https://xata.io/docs/concepts/limits) page in the docs.

If you are a seed-stage startup, a non-profit, or you were in our Beta program, we can increase the limits further, [contact us](mailto:info@xata.io) and we’d be happy to discuss. You can think of us as your team of DevOps people experts in databases.

More than that, we will be on [discord](https://xata.io/discord) to help you along the way. Make sure to say hi so we can send you some swag when you publish your work.

If you read this far but didn’t try Xata yet, you should definitely [do it now](https://app.xata.io/signup)! 🦋

Starting today, Xata is out of private Beta! After more than a year of work, receiving and addressing feedback from over 700 developers, and stretching our designers thin in the past weeks, we are very excited to show you what we’ve been up to. 

Xata is now Live! 🚀

A post detailing how an app was built on Xata with Next.js 13 using its powerful features.

Searching for retro games with Xata & Next.js 13

Next.js 13 is the latest version of the popular full-stack framework based on React. In this post, we'll take a look at the new data fetching features and how Xata can be seamlessly used with their *next* big version.

As announced at the Next.js Conf, Next.js 13 offers a completely new solution for file-system routing, nested layouts, incrementally streamed rendering, and component-level data fetching with React Server Components (RSC). You can find out more in [the Next.js blog announcement](https://nextjs.org/blog/next-13) or replay the keynote from Next.js Conf 2022.

At Xata, we are very excited about this new data fetching mechanism. It removes even more barriers between server and client code and is built on top of web standards, like the fetch API, and it works out of the box with our TypeScript SDK.

### React Server Components and Suspense for Data Fetching

Prior to Next.js 13, data fetching on Next.js was usually done with `getStaticProps` and `getServerSideProps`, two powerful functions that allowed per page define server-side code that would be serialised and sent over the network to the user’s react code in the browser.

React 18 and Next.js 13 are offering a new way to build and run React Components either on the client or the server. Paired with Suspense for Data Fetching, now your application can safely access backend resources or fetch data without leaking sensitive data.

Moreover, taking advantage of the new nested layout mechanism allows for instant rendering of page parts that do not specifically require data. Your application can show a loading state for parts of the page that are fetching data and reduce the time to load the content.

### How the Xata SDK can be used with Next.js 13

At Xata we are building a serverless data platform that is an excellent companion for Next.js. 

Since the Xata SDK is built on top of the same standards as the new data fetching mechanism from Next.js 13, you can already use them together and it will work out of the box. Let’s see an example!

```tsx
const xata = getXataClient();

export default async function Blog() {
  const posts = await xata.db.posts.getAll();

  return (
    <div>
      <h1>Blog posts</h1>

      {data.map((post) => (<h2>{post.title}</h2>))}
    </div>
  );
}
```

Simple, right? As you can see, by only defining the Xata SDK client outside of the server component, you can run any async query or function and the results will be available to be used in the component.

In a future update, we will expose the new cache strategy and data revalidation properties, so that when using Xata and Next.js you can take full advantage of static site generation, server-side rendering, and incremental static regeneration with the new `use` hook.

Stay tuned for our documentation updates that will showcase the newest way of fetching and rendering data with Next.js.

### Conclusion

We are very proud of our partnership with Vercel, sharing a common vision on how modern and performant web applications can be built.

If you want to discuss more about the newest features in Next.js or you have already tried the new data fetching mechanism come to our [discord](http://xata.io/discord) and chat with us!

Next.js 13 is the latest version of the popular full-stack framework based on React. In this post, we'll take a look at the new data fetching features and how Xata can be seamlessly used with their next big version.

The future of data fetching with Next.js 13

With a database access key being sent out from the browser, your database is readable and writable by anyone. They may read the key and therefore use the key to create malicious connections to your database. These actions (attacks) are called **exploits**, and the data retrieved from such exploits are called **leaks**. Yes, like the famous “data leaks” we hear about in the news. Whether your leaked app makes to the news or not will depend on how big and influential your user-base is, but the problem is still the same. Someone with the key can even delete all of your data if they so choose.

When building an app, a common pattern is to build an API (Application Program Interface) that will work as a bridge between your database and the client-side app. Said API will expose a certain number of endpoints depending on how it is designed. For example:

![Diagram of App-API-DB architecture](https://i.imgur.com/NgpZvIp.jpg)

This architecture is not only common, it's a requirement. It happens because a database requires **access keys**. As anything valuable, user data must be stored behind locked doors. Wherever the query originates from, must add these **access keys** (often referred to as **access tokens** as well) to the request. This is why any communication with a database must happen in a **private connection**. 

A browser is a **public** interface. Whatever data and code an app has in the browser is **exposed the public**. Now connect the dots: if everything on the browser is available to the public, and a query (request) goes from the browser straight to the database, what happens to the secure key it must carry to actually retrieve the data? You guessed it: the key is readable, as said in security contexts, it is **exposed**. 

## Querying Xata

To make things more ergonomic, Xata has a [Software Development Kit (SDK)](https://xata.io/docs/sdk/getting-started) we call [client-ts](https://github.com/xataio/client-ts). Under the hood, there are many best practices we enforce. It also generates TypeScript types for your data request and responses based on your schema - that is the Object Relational Mapping (ORM) side of our SDK. 

But arguably, the most important aspect of our SDK is that it will securely query Xata using your environment variables (`process.env`) the `XATA_API_KEY`, which is the key required to query your database. There multiple different ways to expose this environment variable to your running app, for example:

- when calling the script:

```bash
XATA_API_KEY=my_api_key node ./script/query.js
```

- more commonly, via the `.env` configuration file:

```bash
XATA_API_KEY=my_api_key
```

When deploying your app, you must find out how to define Environment Variables to the deployment script. Each PaaS (Platform as a Service) has a different way of doing that. Have a look for examples at [CircleCI](https://circleci.com/docs/env-vars/), [Cloudflare Workers](https://developers.cloudflare.com/workers/platform/environment-variables/), [Netlify](https://docs.netlify.com/configure-builds/environment-variables/), or [Vercel](https://vercel.com/docs/concepts/projects/environment-variables).

The `process.env` namespace is not available in browser environments so if using the SDK, you will experience *friction* when attempting to query your Xata database from the browser. **This is intentional and by design.** Essentially, `process.env.XATA_API_KEY` will be `undefined`,  and the SDK will not be able to fire your request (thus preventing you from leaking access to your database), and you will see a runtime error like:

```bash
error Error: Option apiKey is required
    at XataClient.parseOptions_fn (index.mjs?4597:3090:1)
    at new _a (index.mjs?4597:3053:1)
    at XataClient._createSuperInternal (_create_super.mjs?05d7:12:1)
    at new XataClient (xata.codegen.ts?b4b0:32:24)
    at getXataClient (xata.codegen.ts?b4b0:42:14)
    at eval (stepone.js?34b1:101:47)
```

If on a Node.js environment, this means the SDK cannot read your API key. You can solve this by using one of the methods mentioned above to expose your access key to your runtime. If using the `.env` approach, you may need a package like [dotenv](https://www.npmjs.com/package/dotenv) to read those values.

If on a browser environment, this error **********************************************should not be “fixed”**********************************************. As explained in the first section of this article: exposing your Xata API keys to a client-side runtime like a browser is a huge security issue. For more information, read our documentation about [Securely Talking to Xata](https://xata.io/docs/quickstart/updating-data-with-xata#securely-talking-to-xata). 

## Creating Apps and Connecting to Xata

The path of least resistance to create an app and connect it to Xata is to use a full-stack framework such as [Next.js](https://nextjs.org/docs), [Remix](https://remix.run/docs/en/v1), [Nuxt](https://nuxtjs.org), [Gatsby](https://www.gatsbyjs.com/docs/), and others. Such apps offer a way of creating **serverless functions** within them. In Next.js for example, any files in `./pages/api` contain these functions. But what are they? In a nutshell serverless functions are pieces of code that will run only on the server, allowing developers to query their database there and thus remaining close to the app logic and more importantly, ***out of the unsafe browser environment.***

When creating a pure client-side app ([Astro](http://astro.build), [Eleventy](https://www.11ty.dev/), [Qwik](https://qwik.builder.io/), [React](https://beta.reactjs.org/), [Solid](https://www.solidjs.com/), [Svelte](https://svelte.dev/), [Vue](https://vuejs.org/), etc.), the database request will not be able to originate from the app. Instead, refer to the initial architecture we outlined in this post: first hit an API, then the API connects to the Database. This keeps your API keys secret. 

### What are Serverless Functions?

Different than a long-standing server that will continue to run even when there are no requests coming in or out, a serverless function will cease activity and go into “sleep” whenever its execution is finished. This makes serverless functions a perfect use-case to bridge connections between your app and your database.

Additionally, serverless providers like [AWS Amplify](https://aws.amazon.com/amplify/), [Cloudflare Workers](http://workers.cloudflare.com), [Netlify](http://netlify.com), and [Vercel](http://vercel.com) will provide developers with a no-obstacle deployment. You define your function handler (most receive 2 parameters: `request` and `response`). 

## Connecting a Serverless Function to Xata

Creating a serverless function is basically as straightforward as creating a JavaScript file.

```jsx
// In Next.js, this would be ./pages/api/get-data.ts

export default function handler (request, response) {
  // fetch stuff from Xata
  const data = await xata.db.myDb.getMany();
  response.end(data.records);
}
```

To understand how to create one, and how to design your API, we have a [Vercel Serverless Function Sample](https://github.com/xataio/examples/tree/main/apps/starter-vercel-serverless-functions) in our example repository. Do not forget to create and add your API Keys to the deployment routines within your dashboard, otherwise the Xata Client will not have a key to fire requests with.

For a more involved example, refer to the [Quickstart in our documentation](https://xata.io/docs/quickstart/updating-data-with-xata).

## Conclusion

We hope this has made it clearer how to interact with your Xata database from within and outside your apps. Please feel free to create an issue in our [examples repository](https://github.com/xataio/examples) if there is a specific use-case that you would like us to prioritize in showing how to address. 

Additionally, we cannot stress it enough that **we love open-source** and there is some swag waiting for everyone who ships a community sample app, or creates a PR to contribute to any of our projects (examples included!).

Creating a secure connection between an application and a Xata database. How to fetch data from pure client-side apps without exposing your security keys.

Securely Querying Your Database on Xata (and why browser-based requests are not secure)

At Xata, we are currently in closed beta. To support this, we want to have the best documentation we can. This, of course, is easier said than done, so we are experimenting by moving things around, listening to user feedback, and so on.

Our documentation is written in [Markdown](https://www.notion.so/A-Journey-to-Bulletproof-Links-in-Documentation-f108c0ed0acd464bb1e6310519d19e37), served by [Next.js](https://www.notion.so/A-Journey-to-Bulletproof-Links-in-Documentation-f108c0ed0acd464bb1e6310519d19e37). This has a few advantages:

- we have total freedom in the design,
- we can fetch Markdown documents from GitHub to keep the documentation of a given product close to its code,
- we can have custom components: graphs, Twitter cards, etc.

So far, we had no way to make sure that links between pages were working and that they didn’t just 404. After moving things around, even after checking and rechecking everything manually… **we had 28 broken links in our production documentation.**

![This is fine meme](/images/blog/bulletproof-docs-links/Untitled.png)

This is fine, we all make mistakes, but let’s learn from this together!

## The Plan

The first step is to list our requirements so we have our goal well defined. We want quick feedback if something gets broken, so we can fix it before merging to production. For that, we need to be able to run the docs consistently in different environments:
   - local
   - preview/ staging
   - production

With this in mind, one constraint immediately becomes clear to us: this would be way easier if we don’t have a link checker that needs to run on a server.

If we want to avoid a server, we need to extract the information:

- the internal links in my markdown files
- the directory structure.

## The Proof of Concept

To achieve our final goal, we need a reliable way to extract links from our existing documentation and see if the link *actually goes somewhere*. This is a lot of files, with a lot of links. To simplify our problem, let’s omit the real markdown documents for now and focus on the link extraction part.

> Our docs are at https://xata.io/docs by the way, if you’d like to explore them and use Xata yourself.
> 

1. First, we need to read, parse, and understand a Markdown document. Our parser of choice is  `unified`. It returns an **Abstract Syntax Tree (AST)**. Let’s give it a Markdown document and see what happens.

```tsx
import { describe, expect, it } from "vitest";

import { unified } from "unified";
import remarkParse from "remark-parse";
import remarkStringify from "remark-stringify";

describe("markdown link checker", () => {
  it("should extract all links from a given document", async () => {
    const document = `
# Title
			
this is a paragraph with [a link](/rest-api/get)

## Another title

and another [link](/rest-api/delete)
`;

    await unified()
      .use(remarkParse)
      .use(() => {
        return function transform(tree) {
          console.log(tree);
        };
      })
      .use(remarkStringify)
      .process(document);

    expect(true).toEqual(false);
  });
});
```

When we `console.log` the output, we get this:

![Untitled](/images/blog/bulletproof-docs-links/Untitled%201.png)

And just like this, we can see our markdown content in a structured way, cool no? (we can also use [https://astexplorer.net/](https://astexplorer.net/) to see explore our AST)

2. Extract the links

To do this, we can use `unist-util-visit`, which gives us a very convenient and type-safe way to “visit” our AST.

What does “visit our AST” mean? Every time we have a node with  `type === "link"`, even very deep inside `children`, a function is called with this node as argument! This function is called a **visitor function**. Good news, this is exactly what we want! 😀

Instead of using our previous raw  `tree` object and iterating through every node looking for those `link` objects, we can write this:

```tsx
import { describe, expect, it } from "vitest";

import { unified } from "unified";
import { visit } from "unist-util-visit";
import remarkParse from "remark-parse";
import remarkStringify from "remark-stringify";

describe("markdown link checker", () => {
  it("should extract all links from a given document", async () => {
    const document = `
# Title
			
this is a paragraph with [a link](/rest-api/get)

## Another title

and another [link](/rest-api/delete)
`;

    const links: string[] = [];

    await unified()
      .use(remarkParse)
      .use(() => {
        return function transform(tree) {
					// Let's extract every link!
          visit(tree, "link", (linkNode) => {
            links.push(linkNode.url);
          });
        };
      })
      .use(remarkStringify)
      .process(document);

    expect(links).toEqual(["/rest-api/get", "/rest-api/delete"]);
  });
});
```

That’s essentially all we need for a proof of concept (POC), we are now more confident that we should be able to extract every link from our documentation. Let’s try with some real data!

## The Real Deal

First, we need a way to load all our Markdown files. Luckily, [npm](https://www.notion.so/A-Journey-to-Bulletproof-Links-in-Documentation-f108c0ed0acd464bb1e6310519d19e37) already has a solution for us. 😀

```tsx
import { describe, expect, it } from "vitest";

import { unified } from "unified";
import { visit } from "unist-util-visit";
import remarkParse from "remark-parse";
import remarkStringify from "remark-stringify";
import { getAllFiles } from "get-all-files";
import { readFile } from "fs/promises";

describe("markdown link checker", async () => {
  for await (const filename of getAllFiles("./content")) {
    if (!filename.endsWith(".md")) continue;
    it(filename, async () => {
      const document = await readFile(filename, "utf-8");
      const links: string[] = [];

      await unified()
        .use(remarkParse)
        .use(() => {
          return function transform(tree) {
            visit(tree, "link", (linkNode) => {
              links.push(linkNode.url);
            });
          };
        })
        .use(remarkStringify)
        .process(document);

      expect(links).toEqual([]);
    });
  }
});
```

We are iterating on every file from the `/content` directory and reusing our previous logic, replacing the `document` with our actual markdown content!

And… it works! Not useful yet, but we are going in the right direction!

![Untitled](/images/blog/bulletproof-docs-links/Untitled%202.png)

This is also a nice opportunity for us to see what links we actually have! And we can already spot some patterns that we need to deal with as external links (example: [`https://stackoverflow.com/questions/4423061/how-can-i-view-http-headers-in-google-chrome`](https://stackoverflow.com/questions/4423061/how-can-i-view-http-headers-in-google-chrome)) and links with anchors (example: `/cli/getting-started#code-generation`)

First, let’s exclude external links, they are out of our scope since we’re not interested in *other people’s 404s.* We may add this later since we don’t want to link to broken content in the longer term, but for now **scope hammering keeps us focused**.

```tsx
if (linkNode.url.startsWith("/")) {
  links.push(linkNode.url);
}
```

Let’s improve the output of our test: we want one `describe` per Markdown document, and one `it` per link. This is our target output (for now):

![Untitled](/images/blog/bulletproof-docs-links/Untitled%203.png)

So we have:

```tsx
describe(filename, async () => {
	// ...
	links.forEach((link) => {
        it(`should have "${link}" defined`, () => {
          expect("todo").toBe("done");
        });
      });
	})
})
```

And to finish this part, we need to check if the file exists, `fs.existsSync` will do the job:

```tsx
import { describe, expect, it } from "vitest";

import { unified } from "unified";
import { visit } from "unist-util-visit";
import remarkParse from "remark-parse";
import remarkStringify from "remark-stringify";
import { getAllFiles } from "get-all-files";
import { readFile } from "fs/promises";
import { existsSync } from "fs";

describe("markdown link checker", async () => {
  for await (const filename of getAllFiles("./content")) {
    if (!filename.endsWith(".md")) continue;
    describe(filename, async () => {
      const document = await readFile(filename, "utf-8");
      const links: string[] = [];

      await unified()
        .use(remarkParse)
        .use(() => {
          return function transform(tree) {
            visit(tree, "link", (linkNode) => {
              if (linkNode.url.startsWith("/")) {
                links.push(linkNode.url);
              }
            });
          };
        })
        .use(remarkStringify)
        .process(document);

      links.forEach((link) => {
        it(`should have "${link}" define`, () => {
          expect(existsSync(`./content${link}.md`)).toBeTruthy();
        });
      });
    });
  }
});
```

### Dealing with Anchors

Now, this is where, personally, I started by removing the anchor and called this done.

```tsx
it(`should have "${link}" define`, () => {
  const [path] = link.split("#");
  expect(existsSync(`./content${path}.md`)).toBeTruthy();
});
```

After a coffee and some time to think, I realized—we can do better! And also, I’m sure we have some broken anchors. (Spoiler alert, I found one 😅)

The first problem is that we need to have a dictionary of all anchors per document. So far, our AST logic is in middle of my unit test. Time to refactor!

Let’s extract our current logic into a function:

```tsx
/**
 * Retrieve useful information from a markdown file.
 *
 * @param path file path
 */
async function parseMarkdown(path: string) {
  const document = await readFile(path, "utf-8");
  const links = new Set<string>();

  await unified()
    .use(remarkParse)
    .use(() => {
      return function transform(tree) {
        visit(tree, "link", (linkNode) => {
          if (linkNode.url.startsWith("/")) {
            links.add(linkNode.url);
          }
        });
      };
    })
    .use(remarkStringify)
    .process(document);

  return {
    links,
  };
}
```

Let’s analyze what this method is doing in plain English:

- We’re taking a `path: string` as input
- We’re reading the content of the file that has this `path`
- We’re instatiating a `links` set, ready to be filled
- We have our AST visitor that:
    - on each node that fulfills the condition `type === "link"`
    - if the url starts with `/` (so it’s an internal link), we add it to `links`
- We return the consolidated `links` in an object

Now, we can extract the `anchors` following the same pattern

```diff
async function parseMarkdown(path: string) {
   const document = await readFile(path, "utf-8");
   const links = new Set<string>();
+  const anchors = new Set<string>();

   await unified()
     .use(remarkParse)
		 .use(() => {
		     return function transform(tree) {
		       visit(tree, "link", (linkNode) => {
		          if (linkNode.url.startsWith("/")) {
		            links.add(linkNode.url);
		          }
		        });
         });
+        visit(tree, "heading", (headingNode) => {
+          if (headingNode.children[0].type === "text") {
+            anchors.add(kebab(headingNode.children[0].value));
+          }
+        });
       };
     })
     .use(remarkStringify)
     .process(document);

   return {
+    anchors,
     links,
   };
 }
```

In Markdown, an anchor is any title converted to kebab case. For example, if I have a title `## Create Table` in `/getting-started`, I can point to `/getting-started#create-table`

Time to try our brand new function! Remember, we are just doing small steps—no need to run.

```tsx
import slash from "slash";

type FilePath = string;

describe("markdown link checker", async () => {
  // Collect all the data
  const pages = new Map<
    FilePath,
    { anchors: Set<string>; links: Set<string> }
  >();

  for await (const filename of getAllFiles("./content")) {
    if (!filename.endsWith(".md")) continue;

    pages.set(
      slash(filename)
        .replace(/^\.\/content/, "") // remove `/content`
        .slice(0, -3), // remove `.md`
      await parseMarkdown(filename)
    );
  }

  console.log(pages);
});
```

This test yields this output:

![Untitled](/images/blog/bulletproof-docs-links/Untitled%204.png)

Again, since we’re not worried about the entire problem, we can take our time to clean and prepare our object for the next step:

- Use `slash` so we don’t have those nasty backslashes (windows… my old friend…)
- Remove the `/content` and `.md` from my `path` so we have the same pattern as `anchor`

We can also spot a little problem here, did you see it? The `sidebar-position-5\nsidebar-label-api-keys` entry is not a heading! It’s a `yaml` node and needs to be parsed with the `remarkFrontmatter` plugin!

The source:

```tsx
---
sidebar_position: 5
sidebar_label: API Keys
---
# API Keys
```

We need to add `remarkFrontmatter` to the stack:

```tsx
import remarkFrontmatter from "remark-frontmatter";

// ...
await unified()
    .use(remarkParse)
    .use(remarkFrontmatter) // <- Just here
    .use(() => { /* ... */ } )
```

## The Wrap Up

This is the final version (the one used in our actual documentation repository!)

```tsx
import { describe, expect, it } from "vitest";
import { readFile } from "fs/promises";
import { unified } from "unified";
import { visit } from "unist-util-visit";
import remarkParse from "remark-parse";
import remarkStringify from "remark-stringify";
import remarkFrontmatter from "remark-frontmatter";
import { getAllFiles } from "get-all-files";
import { kebab } from "case";
import slash from "slash";

type FilePath = string;

describe("markdown link checker", async () => {
  // 1. Collect data from `/content`
  const pages = new Map<
    FilePath,
    { anchors: Set<string>; links: Set<string>; isRef: boolean }
  >();

  for await (const filename of getAllFiles("./content")) {
    if (!filename.endsWith(".md")) continue;

    pages.set(
      slash(filename)
        .replace(/^\.\/content/, "") // remove `/content`
        .slice(0, -3), // remove `.md`
      await parseMarkdown(filename)
    );
  }

  // 2. Generate unit tests
  Array.from(pages.entries()).forEach(([page, def]) => {
    if (def.links.size === 0) return;
    describe(page, async () => {
      Array.from(def.links.values()).forEach((link) => {
        it(`should have ${link} define`, async () => {
          const [path, anchor] = link.split("#");
          expect(pages.has(path)).toBeTruthy();
          if (anchor && !pages.get(path).isRef) {
            expect(pages.get(path).anchors.has(anchor)).toBeTruthy();
          }
        });
      });
    });
  });
});

/**
 * Retrieve useful information from a markdown file.
 *
 * @param path file path
 */
async function parseMarkdown(path: string) {
  const document = await readFile(path, "utf-8");
  const links = new Set<string>();
  const anchors = new Set<string>();
  let isRef = false; // `true` if the markdown content is `See …`

  await unified()
    .use(remarkParse)
    .use(remarkFrontmatter)
    .use(() => {
      return function transform(tree) {
        visit(tree, "link", (linkNode) => {
          if (
            linkNode.url.startsWith("/") &&
            !linkNode.url.startsWith("/api-reference")
          ) {
            links.add(linkNode.url);
          }
        });

        visit(tree, "heading", (headingNode) => {
          if (headingNode.children[0].type === "text") {
            anchors.add(kebab(headingNode.children[0].value));
          }
        });

        visit(tree, "paragraph", (paragraphNode) => {
          if (
            paragraphNode.children.length === 1 &&
            paragraphNode.children[0].type === "text" &&
            paragraphNode.children[0].value.startsWith("See ")
          ) {
            isRef = true;
          }
        });
      };
    })
    .use(remarkStringify)
    .process(document);

  return {
    anchors,
    links,
    isRef,
  };
}
```

Few highlights:

- We don’t need `fs.existsSync` anymore, we already have everything we need!
- We ignore `/api-reference` links, they are generated from our an external document at build time, so it’s out of our current scope.
- We added a last concept of `isRef`. This is again a very specific edge case that I don’t want to deal with, we have a part of our documentation fetched from GitHub with the following pattern:
    
    ```markdown
    See [link]
    ```
    

During this journey, I spotted 1 broken link and 1 broken anchor, but more importantly, I’m way more confident that we will not ship broken links in our documentation anymore. 

**This is what’s most important to us as developers: confidence in our code.** Through these tests and this approach, we have a little more of that with our documentation.

## The Conclusion

I hope you learned as much as I did! Please borrow ideas & code from this article and iterate from this. Unit tests are sometimes seen as something that slows us down or a boring thing to do, but I personally had a lot of fun playing with `unified` and Markdown’s AST, and I’m 100% convinced that this was not a waste of time—you can’t imagine how much time I spent on the documentation clicking every link, *and I still missed 2 of them!* This 100-line implementation is way more efficient than my manual QA 😅.

Happy hacking!

How we permanently fixed the problem of broken internal links in our documentation.

A Journey to Bulletproof Links in Documentation

At Xata we strive to be the **serverless database with the best developer experience** (DX) in the world. This is an ambitious goal and all of us are fully aware of that, but we have our vision and I can say confidently we are putting in the work to get there. Today we share yet another step towards this goal: we are opening our [Xata Discord Server](https://xata.io/discord) to everyone interested in joining in. “What does a discord server have do with DX?”, you may be wondering. I am glad you asked.

## The Vision

Having good DX means more than easy to use tools and readable code to us. It means being close to our users, it means knowing them, it means _being there_. Feedback like this from [selbekk](https://twitter.com/selbekk) gets us celebrating for days:


![Selbekk on twitter: "Hey let me just say - Xata is a great product and all, and the developer experience is really good. However, the very best thing about Xata isn't the product at all - it's the people helping me out with my onboarding experience, answering questions around the clock and just providing me with a great experience all around. Thanks"](/images/blog/discord-community/selbekk-tweet.png)

From the Developer Relations team to the CEO, we are all there for you. Engaging, listening to your feedback and learning from how you use our product and what you want to build with us. We believe only our users can tell us what the _best serverless database DX in the world_ actually feels like, and we are here for it.

Creating a community also enables us to live up to another of our standards: **inclusiveness**. We want everyone to feel they belong in tech. And this server is our little corner where we will ensure everyone has the emotional safety to speak and be heard, to participate, to teach, and to learn.

## Sharing Knowledge

Within our server, sharing the knowledge you have (or just acquired) has a main role. Because it is easy to miss information in the endless scroll of channels where different topics intertwine and people get overwhelmed, we have created a set of channels with the main goal of encouraging people to participate and to allow what is said to linger for decent time. No FOMO (fear of missing out) around there.

We have different channels for:

- **showcasing** what you built
- ask for **help** for assistance in whatever you are building
- share tips via our twitter hashtag `#XataTips` (or a nice vibe with its sibling: `#XataVibes`)
- or maybe just **request access** because you had an awesome idea and we're in closed beta still

## Short Feedback Loop

On the first section, I mentioned we keep our ears up for users' feedback. Our discord server amplifies that in the way that we have announcement channels, where you will be able to hear first what we are working on and releasing. And, because we are all there, you can offer feedback in real-time as we share, or as we are implementing things.

- Do you want `.csv` export? Start a thread there.
- Is there a data type missing for your requirements? We want to know.

Drop a message, start a thread, let us know.

## Co-working space

As the cherry on top, we are a remote-first team. We know how the pandemic made everyone struggle and some are still in need of live interactions. So we have a special voice channel for **co-working**. Jump in the channel and someone may soon join in and chat for a bit. Maybe you're flirting with becoming a streamer, or you want some rubber ducking, or even you just want some coffee-break/watercooler kind of chat. Anything goes.

## Join us. Join us now!

In case you aren't familiar with discord, it's free. There isn't any information required, you open your account with an email and you can join our server with [this invitation](https://xata.io/discord). If something doesn't work well for you, reach out to us on [twitter](https://twitter.com/xata) or via `info[at]xata.io`. We'll do our best to help you sorted out. Say "hi" when you get there!


Come over and join the Xata Serverless Database Community on discord. It is a free and inclusive space for us to know each other better and learn together! We are also eager to learn from your experiences with our products.

Xata Serverless Database Community

When showing [Xata.io](http://xata.io) (currently in private Beta) to people and saying that it is a “serverless database”, we occasionally get confused looks: how can a database be serverless? Doesn’t it need to store the data _somewhere_? This is a fair question because “serverless”, being the buzzword that it is, doesn’t have a definition that everyone agrees on. 

This is why we thought it’d be worth writing a blog post to document _our_ definition of a serverless database using straightforward language, in an attempt to bring a bit of clarity to the world.

First of all, hat tip to SQLite who was using the [term](https://www.sqlite.org/serverless.html) “serverless database” since before it was a buzzword, to mean something quite different: a database engine that runs in the process of the application. Rest assured, we’re talking about the buzzword serverless here. We’re Venture Capital (VC) funded after all. This version of “serverless” means a service–which _definitely_ uses servers–but those servers are abstracted away as much as possible by the said service.

## No Servers (to manage)

Yes, there are servers involved in our service. No, you don’t need to know about them. You don’t need to care how many of them there are, how much RAM they have, or if there are any low disk-space alerts raised recently, because those are totally and solely our problem (ugh). 

We’re taking the definition further than that to also include that you shouldn’t have to configure things like clusters, instances, nodes, backups, availability zones, monitoring, or what have you. Again, **those are our things to worry about**.

As a user, what you get is series of API endpoints that you can use to insert and update data, plus a set of APIs for querying, searching, and aggregating data. That’s it. Is there a fine print that comes with that? Oh yes, of course there is, and we’ll talk about that in the rest of the blog post.

Small side note: while not necessarily important for the definition of a serverless database, our API speaks HTTP and is therefore connectionless, it’s a lot easier to use from serverless functions because there’s no connection management to worry about.

## No Upgrades (that you have to worry about)

Databases, like most software, need occasional upgrading. The upgrading of databases in particular is usually annoying, dangerous, and can cause downtime. Some DBaaS (Databases as a Service) will take care of at least minor version upgrades for you, sometimes at the cost of having a maintenance window.

In our definition of serverless, there are no upgrades. Scratch that—just like with servers, there _are_ upgrades–it’s just that _we_ are the ones that do them. We do both minor and major upgrades in such a way that there’s no observable downtime to your users.

On the other hand, there might be versions of our APIs that you might want to upgrade to so you’re not _totally_ off the hook, but that tends to be a lot simpler by comparison.

## Durability and Availability (and SLAs)

When you send data to a cloud service like AWS’s S3 (Simple Storage Service), you don’t expect it to get lost with any sort of reasonable probability. That means that the service is highly durable. You also expect to be able to read the data at any time. That means that the service is highly available. Behind the scenes, in order to provide high-durability and high-availability, the service needs to store the data in multiple copies on different hard drives, and run servers in multiple availability zones. This also applies to databases.

If the database service is serverless, you shouldn’t have to worry about how many replicas and in which availability zones they are. Instead, as a user, you get an SLA (Service Layer Agreement) which details the durability and availability guarantees the service gives you, and what happens if those guarantees are not held. Usually, you get some money back.

We think high-durability and high-availability are important aspects of a serverless database service, so we include them in our definition and in all pricing plans, including the free tier. 

We’re still working on documenting our limits and SLAs, that’s part of why we’re still in closed beta.

## Performance and Scalability (and how they relate to pricing)

This is the part where we’d claim inter-planetary scale (to one-up our perceived competition) and “blazing-fast” performance for any query. And you’d just skip to the next headline because *of course we’d claim that*.

Instead, let’s have a discussion: what happens if a user sends a series of very inefficient queries quickly and in parallel?

There are database services out there that just don’t allow any queries that *can* be inefficient. That’s great from a service point of view, but their data model tends to be quite restrictive, usually restricted to some sort of key-value store.

Ours is not one of those. We have a pretty rich [data model](https://docs.xata.io/concepts/data-model) which allows complex queries and relations and everything, so there can be inefficient queries simply because they have to check a lot of data. Those inefficient queries are expensive for us to run, which is kind of a problem. In this case, there are two options:

- make the price per query proportionate to the work the query has done
- throttle queries and degrade the service predictably to the agreed capacity

We think both are valid strategies for a serverless database service, but in both cases you need to pay attention to the details. A drawback of the former is that it can become very expensive very quickly when you have a bad query. Some services allow you to set cost budgets, which are ideally checked in real-time. This replaces the risk of surprise bills with the risk of downtime when the budget limit is hit.

We are trying the second approach, which has its own challenges, but we think it’s ultimately the better model for the users. This model is less used today, but DynamoDB, one of the first serverless DBs, works like that. You can reserve a certain capacity, and you get throttled if you surpass it. In a way, it simulates what you would get from a “serverful” setup: if you haven’t allocated enough capacity, your performance will be degraded, but it won’t suddenly break on you. The difference is that we can make it a lot easier to add capacity or configure auto-scaling.

## Scaling to Zero (mostly)

A very cool aspect of the original serverless service, AWS Lambda, is that it scales down to zero in terms of pricing. That means that if a function is not used, it takes (next to) no resources and therefore costs nothing. This is great for personal projects but also for particular use cases even in the enterprise. This concept mostly translates to databases, but not quite–because even if the database receives no requests at all, it still needs to store the data.

In our definition of a serverless database, the compute costs should scale down to zero and costs should be incurred only if a lot of data is stored.

But there’s one more aspect to it: the infamous cold starts. Databases are typically not as quick to provision and boot as lambda functions. Some DBaaS will even suspend your instances after a few days of inactivity and you need to manually bring them up. 

In our definition, cold starts are very small and databases don’t get suspended after a period of inactivity that is measured in days. We’re hedging here a bit because we might have to eventually suspend databases as well, but we’d only do it after a very long period of inactivity: think years, not days.

## Summary (and no silver bullet)

Our definition of serverless databases can be summarised like this: **we take care of as many hard problems as possible so you don’t have to**. 

That includes scaling, upgrades, replication, schema migrations, backups, monitoring, and so on. But you don’t have to pity us for all those problems, we kind of like this type of work, and to be fair, we’re charging you for it–just less than what it would cost you to hire your own DB experts. 

That’s the value proposition of serverless databases as we see it, and we think it’s worth the hype: you’ll be able to run a business the scale of the next Airbnb, Uber, or Netflix, without DB experts and with a lot fewer “platform engineers” in general.

Is it a silver bullet, though? Do we take care of _all the problems_ that you can have with a database? Nope and nope.

For example, a bad data model or code that produces very inefficient queries will be either expensive for you or will feel sluggish for your users. Or maybe both. So it’s much better to fix the code. Also, our service, like any other, has various limitations, restrictions, and SLAs. What we promise is to document these clearly and explain them in plain language.

Being a serverless database and taking care of all of these problems for you is actually only half of Xata’s value proposition. The other half is about providing a top-notch developer experience and simpler developer workflows. But we’ll need to talk about that later, when we’re out of the closed beta. Until then, you can register for early access or follow us on [twitter](https://twitter.com/xata_io) for good vibes and so we can keep you in the loop.

What is a Serverless Database? There isn't a definitive definition that everyone agrees to, so this post documents our own definition, in an attempt to bring a bit of clarity to the world.