Batch checking

Run a folder of documents through the API with a five-wide pool, honoring the concurrency and hourly limits, and collect one row per document.

The account may have five checks in flight at once and a key may start 300 checks an hour. A batch runner that respects both is a small pool: keep up to five ids in flight, poll them, and submit the next document as each one finishes.

batch.ts

import { readdir, readFile } from 'node:fs/promises';
import path from 'node:path';
import { Silvertext } from './silvertext'; // from the Node.js how-to

const POOL = 5;
const st = new Silvertext();

interface Row {
  file: string;
  id?: string;
  status: string;
  plagiarism?: number;
  aiContent?: number | null;
  grammar?: number;
  error?: string;
}

export async function checkFolder(dir: string): Promise<Row[]> {
  const files = (await readdir(dir)).filter((f) => /\.(txt|md|docx|pdf)$/i.test(f));
  const rows: Row[] = [];
  const queue = [...files];
  const inFlight = new Map<string, Row>();

  async function submitNext() {
    const file = queue.shift();
    if (!file) return;
    const row: Row = { file, status: 'submitting' };
    rows.push(row);
    try {
      const bytes = await readFile(path.join(dir, file));
      const { id } = await st.createCheck(
        { file: bytes.toString('base64'), filename: file, options: { citations: false } },
        `batch:${dir}:${file}`, // idempotent: rerunning the batch never double-charges
      );
      row.id = id;
      row.status = 'queued';
      inFlight.set(id, row);
    } catch (err) {
      row.status = 'error';
      row.error = (err as Error).message;
      await submitNext();
    }
  }

  while (inFlight.size < POOL && queue.length) await submitNext();

  while (inFlight.size) {
    await new Promise((r) => setTimeout(r, 3000));
    for (const [id, row] of inFlight) {
      const check = await st.getCheck(id, { view: 'summary' });
      row.status = check.status;
      if (['done', 'rejected', 'failed', 'cancelled'].includes(check.status)) {
        inFlight.delete(id);
        if (check.status === 'done' && check.scores) {
          row.plagiarism = check.scores.plagiarism;
          row.aiContent = check.scores.aiContent;
          row.grammar = check.issueCounts?.grammar;
        } else {
          row.error = check.rejection?.message ?? check.status;
        }
        await submitNext();
      }
    }
  }
  return rows;
}

What to watch

Credits first. GET /v1/me/usage before a big batch; multiply the folder's word count by 0.6 to estimate the credits it needs. A 402 quota_exceeded mid-batch is recoverable (buy a pack and rerun; the idempotency keys skip the documents already checked), but it is better to know up front.
The hourly rail. 300 starts an hour is 5 a minute; a pool of five that finishes each check in about a minute sits right at it. The client's retry loop honors Retry-After when you cross it.
Polling volume. Five in-flight checks polled every 3 seconds is 100 requests a minute, near the 120 limit with nothing to spare for submits. Poll every 4 to 5 seconds in a batch, or poll the oldest check only and sweep the rest when it finishes.
Rejections are per document. A scanned PDF with no readable text or a document in an unsupported script rejects on its own; the batch continues.
Clean up. Delete reports you have exported (DELETE /v1/checks/{id}) if your product keeps its own copy.

Batch checking

What to watch

On this page