Build and refresh live-web datasets with local Hermes + Codex OAuth agents.

BigSet turns a plain-English request into a structured dataset. This fork keeps the original BigSet OpenRouter + TinyFish path, and adds a local-first Hermes mode where one Hermes Agent instance replaces both external AI/search services.

Example prompt:

5 popular open-source database engines with license and first release year

BigSet infers a schema, researches public web sources, writes verified rows into Convex, and exports CSV/XLSX. Datasets can be refreshed on a schedule.

This branch is packaged as a public, reproducible reference for running BigSet with Hermes Agent and Codex OAuth:

• BigSet turns natural-language dataset ideas into typed, refreshable datasets.
• Hermes Agent supplies the model/runtime layer through its OpenAI-compatible API server.
• Hermes can use the openai-codex provider, so the same ChatGPT/Codex OAuth account that powers Hermes can power BigSet research without putting OAuth tokens in BigSet.
• The agent-led bigset skill keeps prompts bounded and source-verifiable before population starts.
• Completed datasets can be used in the UI, exported as CSV/XLSX, rendered into HTML/PDF reports, and emailed to yourself or a client with your own SMTP credentials.

The current explainer video source and rendered media live in artifacts/bigset-hermes-x-video/.

The preferred user workflow is agent-led, not manual prompt-writing.

When a user says “make a BigSet,” “create a new dataset,” or “build a dataset for this client,” load the Hermes skill named bigset and run a short discovery pass before creating the dataset prompt. The agent should clarify:

1. business goal — what decision/action the dataset should drive;
2. row target — companies, local businesses, venues, events, products, creators, jobs, RFPs, etc.;
3. scope — geography, industry, audience, source constraints;
4. qualifying public signal — event pages, job posts, procurement pages, vendor pages, menus, calendars, social/event pages, etc.;
5. delivery — demo 10 rows or production 25 rows, BigSet link only or export/email.

Then the agent creates the BigSet prompt using the expected structure:

Create a dataset of <10 or 25> <target rows> for <business/user/context>.

Goal:
<business action this dataset should drive>

Scope:
<geography, industry, source types, constraints>

Only include rows with public evidence that <qualification signal>.

Columns:
- <primary_name>
- website
- category / industry / type
- location / geography, if relevant
- <signal column>
- signal_type
- source_url
- signal_date_or_event_date_if_available
- likely_use_case
- confidence_score or outreach_priority_score
- why_it_matters / why_good_fit
- first_outreach_note, if sales-oriented

Rules:
- Return <10 or 25> rows.
- Every row must include a source_url.
- Use public organization/business information only.
- Do not scrape personal emails, private phone numbers, or individual staff contact details.
- Skip rows where the qualifying signal cannot be verified from a public source.
- Scores should be 1-100.
- Keep explanations short and practical.

This avoids prompt drift and produces more useful, source-verifiable datasets than generic one-line prompts.

In hermes mode BigSet does not call OpenRouter, TinyFish, or OpenAI directly. BigSet calls a local OpenAI-compatible Hermes API server. Hermes owns the Codex OAuth token, model selection, and web tooling.

schema inference       -> Hermes /v1/chat/completions -> strict JSON schema
candidate discovery    -> Hermes agent + web_search -> strict JSON entity list
per-entity research    -> Hermes agent + web tools -> strict JSON row data
row writes             -> BigSet insert_row tool -> Convex
refresh runs           -> Hermes agent -> strict JSON update decision

Hermes is an agent endpoint, not a raw tool-calling model. It does not execute arbitrary Mastra client-side tool arrays. BigSet therefore runs deterministic TypeScript orchestration around Hermes calls:

• Hermes returns data only.
• BigSet validates JSON with Zod.
• BigSet performs all inserts/updates itself.
• The existing closure-scoped buildPopulateTools() authorization model remains intact.

Prerequisites:

• Node.js 22+ with npm
• Docker Desktop or compatible Docker engine
• Make
• Hermes Agent installed on the host
• A ChatGPT/Codex account that exposes the Codex model you want to use

Install Hermes if needed:

curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash

Add Codex OAuth credentials and pick the model:

hermes auth add openai-codex --type oauth
hermes model

Choose the openai-codex provider and a Codex-visible model such as gpt-5.5.

Enable/check tools:

hermes tools
hermes doctor

At minimum, Hermes needs working web_search and web_extract tools for BigSet populate/refresh.

Find the active Hermes env file:

hermes config env-path
# or for a named profile:
hermes -p <profile> config env-path

Add:

API_SERVER_ENABLED=true
API_SERVER_HOST=0.0.0.0
API_SERVER_PORT=8642
API_SERVER_KEY=replace-this-local-secret

Start or restart the gateway:

hermes gateway restart
# or foreground:
hermes gateway run

Verify from the host:

curl http://127.0.0.1:8642/v1/models \
  -H "Authorization: Bearer replace-this-local-secret"

cp .env.example .env

Set at least:

LLM_PROVIDER_MODE=hermes
HERMES_API_KEY=replace-this-local-secret
HERMES_BASE_URL=http://host.docker.internal:8642/v1
HERMES_MODEL=hermes-agent

host.docker.internal is required because the backend runs in Docker and must reach the host-side Hermes gateway. If you run the backend directly on the host, use http://127.0.0.1:8642/v1.

make dev

When ready:

Open http://localhost:3500. In Hermes mode the setup screen checks the Hermes endpoint instead of asking for TinyFish/OpenRouter keys.

Create a small bounded dataset first:

5 popular open-source database engines with license and first release year

Expected behavior:

• schema inference returns a compact schema;
• populate target resolves to 5 rows;
• discovery uses a bounded candidate batch;
• rows appear live in the table;
• CSV/XLSX export works.

The app UI exports CSV/XLSX. The repo also includes a public, scriptable report flow for client-ready delivery:

# Replace with a live dataset id from http://localhost:3500/dataset/<id>
node scripts/with-root-env.mjs node scripts/export-dataset-report.mjs \
  --dataset-id <dataset_id> \
  --title "Client-ready BigSet report" \
  --out-dir artifacts/dataset-reports/<dataset_id>

# Optional: email the generated PDF with your own SMTP credentials.
SMTP_HOST=smtp.example.com \
SMTP_PORT=587 \
[email protected] \
SMTP_PASSWORD=app-password-or-secret \
EMAIL_FROM="BigSet <[email protected]>" \
[email protected] \
EMAIL_SUBJECT="Your BigSet dataset report" \
EMAIL_ATTACHMENT=artifacts/dataset-reports/<dataset_id>/report.pdf \
python3 scripts/send-dataset-report.py

This is intentionally externalized into scripts rather than hardcoded into the app: BigSet should never store your ChatGPT/Codex OAuth token or your mail password. Use app passwords, SMTP relay credentials, or your own transactional email provider.

Leave LLM_PROVIDER_MODE unset or set it to openrouter, then provide TinyFish and OpenRouter credentials through setup or .env.

LLM_PROVIDER_MODE=openrouter
TINYFISH_API_KEY=...
OPENROUTER_API_KEY=...

make dev          # start/recover local stack
make down         # stop containers, preserve data
make clean        # stop containers and delete local volumes
make convex-push  # deploy changed frontend/convex functions

make dev bootstraps the local stack: it creates .env if needed, installs frontend/backend dependencies, starts the local keychain bridge, starts Postgres + self-hosted Convex, validates/generates the Convex admin key, deploys Convex functions, starts frontend/backend/Mastra, and streams logs.

If you edit Convex functions under frontend/convex/, run make convex-push. Convex does not hot-reload those functions from the mounted source tree.

This repo is primarily wired for local Docker development. For a public/self-hosted deployment, deploy the same four layers explicitly:

1. Frontend: Next.js app from frontend/.
2. Backend: Fastify app from backend/.
3. Convex: self-hosted Convex plus deployed functions from frontend/convex/.
4. Hermes: a reachable Hermes Agent gateway if using LLM_PROVIDER_MODE=hermes.

Production-style environment:

PROD=1
CLIENT_ORIGIN=https://your-frontend.example
CONVEX_URL=https://your-convex.example
CONVEX_SELF_HOSTED_ADMIN_KEY=...
NEXT_PUBLIC_CONVEX_URL=https://your-convex.example
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=...
CLERK_SECRET_KEY=...
CLERK_JWT_ISSUER_DOMAIN=https://your-clerk-issuer.example

# Hermes mode only:
LLM_PROVIDER_MODE=hermes
HERMES_BASE_URL=https://your-hermes-gateway.example/v1
HERMES_API_KEY=...

Deploy Convex functions after changing schema/functions:

cd frontend
npx convex deploy --url "$CONVEX_URL" --admin-key "$CONVEX_SELF_HOSTED_ADMIN_KEY"

If the Hermes API server is reachable over a network, put TLS and authentication/rate limits in front of it. Never expose an unauthenticated Hermes gateway.

bigset-oauth/
├── frontend/                 Next.js UI + Convex schema/functions
│   └── convex/               Convex schema, authz, rows, model config, seed data
├── backend/                  Fastify + Mastra + Hermes/OpenRouter adapters
│   ├── src/hermes/           Hermes HTTP client and JSON research orchestration
│   ├── src/pipeline/         Schema inference and dataset type contracts
│   ├── src/mastra/           Original workflows/agents/tools and Hermes branch points
│   ├── src/email/            Optional dataset-ready email
│   └── src/analytics/        Optional PostHog wrapper
├── scripts/                  Build/release, verification, report export, and optional SMTP send helpers
├── artifacts/                Public explainer/video artifacts
├── makefiles/                Local Docker workflow
├── HERMES_MODE.md            Operator guide for Hermes/Codex mode
├── IMPLEMENTATION_NOTES.md   Code-level notes on what was replaced
├── docker-compose.dev.yml    Local stack
└── .env.example              Public env template

cd backend && npm run build
cd ../frontend && npm run build
bash scripts/verify-authz.sh

Also run a small Hermes-mode populate test and verify rows appear in the UI. For public video/report changes, verify the HyperFrames artifact and generated report paths too:

cd artifacts/bigset-hermes-x-video && npm run check
node scripts/with-root-env.mjs node scripts/export-dataset-report.mjs --dataset-id <live_dataset_id> --out-dir /tmp/bigset-report-test

AGPL-3.0. See LICENSE.