PlanewardPlaneward Docs
PlanewardPlaneward Docs
Welcome
Sites overviewHeader and footerDark modeDomainsCollectionsRedirectsCrawlers and agentsLinked domainsExternal sites

Developers

REST APIAuthenticationWebhooksMCPEmbedded projectsTask description contentDeveloper hub

API

Sites

Crawlers and agents

Robots rules, Content Signals, /.well-known files, structured data, and Markdown for AI agents.

Open Settings → Crawlers & Agents. These settings apply to the live site as soon as you save. You do not need to publish again.

They only matter on your custom domain. {slug}.planeward.site is always noindex.

robots.txt

Planeward generates /robots.txt. By default it allows everything except /preview, /editor, and /submit, and lists the sitemap on your primary domain when search engines are on (Settings → Search engines).

Add rules to control specific crawlers. Each rule has a user agent and lists of paths to allow and disallow:

User agentDisallowEffect
GPTBot/Keeps OpenAI's training crawler out
*/drafts/Asks every crawler to skip /drafts/

Paths start with /. Up to 20 rules, 50 paths each. A * rule merges into the built-in * group, and the built-in disallows always stay. A named crawler with no paths gets Allow: /. The preview under the form shows the exact file:

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
Disallow: /preview
Disallow: /editor
Disallow: /submit
Disallow: /drafts/

User-agent: GPTBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Content Signals

Content Signals tell crawlers how you allow your content to be used, beyond whether they may fetch it. With Publish Content Signals off, robots.txt makes no statement. Turn it on and set each signal (Search, AI Input, AI Training) to yes or no:

SignalMeans
searchBuilding a search index and showing links and short excerpts
ai-inputUsing pages as input to an AI answer (grounding, RAG)
ai-trainTraining or fine-tuning AI models

They are written into the * group of robots.txt as a Content-Signal: line, and sent as a header on Markdown responses. Signals are a stated preference; crawlers choose whether to honor them.

/.well-known files

Serve small files at /.well-known/{path}, such as security.txt or an api-catalog. Each file has:

  • Path, relative to /.well-known/, like security.txt. Lowercase letters, digits, ., _, -, and /.
  • Content type: text/plain, application/json, or application/linkset+json.
  • Body, up to 32 KB. JSON bodies are checked before you can save.

Up to 10 files.

Structured data

Every page already gets WebSite, a page node, and BreadcrumbList JSON-LD, plus FAQPage when it has an FAQ block, and the collection type on item pages.

Page type

In a page's SEO settings, Page Type sets the page node's schema.org type:

Page TypeUse for
WebPageDefault
ArticleBlog posts, guides, announcements
AboutPageAbout the company or team
ContactPageContact and support pages
CollectionPageIndex pages that list other pages or collection items

An Article page can also take a published date. Its JSON-LD gets headline, datePublished, dateModified (the last publish), and author and publisher set to the site's Organization below, so fill that in first.

Site-wide nodes

Add site-wide nodes here. They go into every page's graph:

  • Organization: name, URL, logo, and profile links (sameAs).
  • Software application: name, category, operating system, and offers (price and currency).

All URLs use your primary domain.

Markdown for agents

Turn on Serve Markdown to AI Agents. Agents that ask for text/markdown get a clean Markdown copy of each page instead of HTML; people still see the normal page. There is no separate URL to share; the agent just sends an Accept header:

curl -H "Accept: text/markdown" https://www.example.com/pricing

The response is text/markdown with Vary: Accept, the Content-Signal header when you set signals, and X-Robots-Tag: noindex so the Markdown copy is never indexed in place of the page. The body starts with front matter (title, description, canonical URL), then the page content, then the page's JSON-LD. Redirects and referral links behave as usual. HTML responses for the same page also send Vary: Accept.

Agent links

Agent links add a Link header to the homepage (HTML and Markdown) so agents can find machine-readable resources without parsing HTML. Each link has a rel and an href (a path or https:// URL):

Link: </.well-known/api-catalog>; rel="api-catalog"

Up to 10 links.

With MCP or the API

planeward-site-crawl-get returns these settings. planeward-site-crawl-update patches them: each top-level key you send (robots, wellKnown, jsonLd, markdownForAgents, agentLinks) replaces the stored value, and omitted keys are kept. See MCP → Sites.

Redirects

Short paths that send visitors to a page or another site, as temporary or permanent redirects.

Linked domains

Keep visitor tracking when people click from your site to your app or checkout on another domain.

On this page

robots.txtContent Signals/.well-known filesStructured dataPage typeSite-wide nodesMarkdown for agentsAgent linksWith MCP or the API