Overview

Introduction

Crawler is a headless content management system designed for speed, clarity, and flexibility. It’s built for modern development workflows, supporting static sites, dynamic apps, and everything in between.

Key features include:

Type-safe content models with automatic validation
GraphQL and REST APIs for flexible data access
Real-time collaborative editing for content teams
Multi-environment deployments and role-based permissions

Quick Example

Here’s how easy it is to get started with Crawler:

import { createClient } from '@scalar/api-client';

export const scalar = createClient({
  projectId: process.env.SCALAR_PROJECT_ID,
  apiKey: process.env.SCALAR_API_KEY,
});

// Fetch content
export async function getBlogPosts() {
  return await scalar.content.findMany({
    model: 'blogPost',
    include: ['author', 'categories'],
  });
}

Core Components

Crawler has several core parts:

Crawler Core

The data and API layer: models, validation, database handling, and schema generation. This is the foundation of your Crawler setup, handling all the backend operations and data management.

Crawler Studio

A lightweight admin UI for managing content with live preview and model editing. The visual interface where content creators and editors can manage your content efficiently with real-time previews.

Crawler Models

Define fields, relationships, validation rules, and access settings. The heart of your content structure. This is where you define how your content is structured, what fields it contains, and how different content types relate to each other.

Crawler CLI

A command-line interface to scaffold content models, sync schema changes, run local servers, and deploy. Ideal for developers who prefer automation over Git-based workflows. The CLI provides powerful tools for managing your Crawler setup programmatically.

Installation

Install Crawler
Section titled “Install Crawler”

Run the installation command:
Terminal window
```
npx create-scalar@latest
```
Configure Your Setup
Section titled “Configure Your Setup”

Follow the prompts to choose your database (PostgreSQL, SQLite, etc.), generate models, and launch the admin panel.
Launch Your CMS
Section titled “Launch Your CMS”

Once installation completes, start your development server:
Terminal window
```
cd your-project-name
npm run dev
```
Your Crawler admin panel will be available at http://localhost:3000/admin

First-Time Setup

Once you’ve installed Crawler, you’ll be guided through the initial configuration process to set up your content models and database connection.

Framework Integration

Crawler works seamlessly with your favorite frameworks:

import { scalar } from '@/lib/scalar';

export default async function BlogPage() {
  const posts = await scalar.content.findMany({
    model: 'blogPost',
    orderBy: { publishedAt: 'desc' }
  });

  return (
    <div>
      {posts.map(post => (
        <article key={post.id}>
          <h2>{post.title}</h2>
          <p>{post.excerpt}</p>
        </article>
      ))}
    </div>
  );
}

<template>
  <div>
    <article v-for="post in posts" :key="post.id">
      <h2>{{ post.title }}</h2>
      <p>{{ post.excerpt }}</p>
    </article>
  </div>
</template>

<script setup>
const { $scalar } = useNuxtApp();

const { data: posts } = await $scalar.content.findMany({
  model: 'blogPost',
  orderBy: { publishedAt: 'desc' }
});
</script>

<script>
  export let data;
</script>

<div>
  {#each data.posts as post}
    <article>
      <h2>{post.title}</h2>
      <p>{post.excerpt}</p>
    </article>
  {/each}
</div>

import { scalar } from '$lib/scalar';

export async function load() {
  const posts = await scalar.content.findMany({
    model: 'blogPost',
    orderBy: { publishedAt: 'desc' }
  });

  return { posts };
}

---
import { scalar } from '../lib/scalar';

const posts = await scalar.content.findMany({
  model: 'blogPost',
  orderBy: { publishedAt: 'desc' }
});
---

<div>
  {posts.map(post => (
    <article>
      <h2>{post.title}</h2>
      <p>{post.excerpt}</p>
    </article>
  ))}
</div>

import { useEffect, useState } from 'react';
import { scalar } from '../lib/scalar';

export function BlogList() {
  const [posts, setPosts] = useState([]);

  useEffect(() => {
    scalar.content.findMany({
      model: 'blogPost',
      orderBy: { publishedAt: 'desc' }
    }).then(setPosts);
  }, []);

  return (
    <div>
      {posts.map(post => (
        <article key={post.id}>
          <h2>{post.title}</h2>
          <p>{post.excerpt}</p>
        </article>
      ))}
    </div>
  );
}

Content Models

Define your content structure with TypeScript-like syntax:

export const BlogPost = {
  title: { type: 'string', required: true },
  slug: { type: 'slug', from: 'title' },
  excerpt: { type: 'text', maxLength: 200 },
  content: { type: 'richText' },
  author: { type: 'reference', to: 'Author' },
  categories: { type: 'reference', to: 'Category', many: true },
  publishedAt: { type: 'datetime' },
  featured: { type: 'boolean', default: false },
};

Creating Content

Use the API to create and manage content programmatically:

// Create a new blog post
export async function createBlogPost(data) {
  return await scalar.content.create({
    model: 'blogPost',
    data: {
      title: data.title,
      slug: data.slug,
      content: data.content,
      author: { connect: { id: data.authorId } },
      publishedAt: new Date(),
    },
  });
}

// Query with filters
export async function getFeaturedPosts() {
  return await scalar.content.findMany({
    model: 'blogPost',
    where: { featured: true },
    include: ['author'],
    take: 5,
  });
}

GraphQL API

Access your content using GraphQL:

query GetBlogPosts($limit: Int = 10) {
  blogPosts(limit: $limit, orderBy: { publishedAt: DESC }) {
    id
    title
    slug
    excerpt
    publishedAt
    author {
      name
      avatar
    }
    categories {
      name
      slug
    }
  }
}