ci: harden description checks — unfilled dropdowns, gameable test plans, non-issue links (#2099)

* ci: harden description checks (dropdown placeholder, how-to-test, link \b)

- issue: flag sections still showing the "-- Please Select --" dropdown
  placeholder (added in #2068) as a single comma-separated line item;
  presence-only checks previously let an un-chosen dropdown pass.
- PR: replace the numbered-step "How to Test" rule with a non-trivial
  content requirement (>=30 chars). The old /\d+\.\s*\S/ rule both
  false-failed prose/code-block test plans and was gamed by an empty
  "1. 2. 3." shell; the message now explains what detail to provide.
- PR: tighten the linked-issue regex to /#\d+\b/ so a hex colour like
  #1a2b3c no longer counts as an issue reference.

---------

Co-authored-by: Povilas Kirna <povilas.kirna@pebble.net>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Povilas Kirna
2026-06-04 08:16:36 +02:00
committed by GitHub
parent 4dc11cfe6b
commit 68eeb7841c
2 changed files with 37 additions and 4 deletions

View File

@@ -91,6 +91,37 @@ module.exports = async ({ github, context, core }) => {
// 'untyped' → only the common body-length check applies. // 'untyped' → only the common body-length check applies.
} }
// ── Unfilled dropdowns ────────────────────────────────────────────────────
// #2068 added a "-- Please Select --" default to every template dropdown, so
// a contributor who never opens the dropdown submits with that literal string
// as the section value. The per-section checks above only verify presence, so
// a placeholder value passes. Scan every section and flag the ones still
// showing the placeholder, as a single comma-separated line item.
const PLACEHOLDER = '-- Please Select --';
const headingRe = /^#+\s+(.+?)\s*$/gm;
const headings = [];
let headingMatch;
while ((headingMatch = headingRe.exec(body)) !== null) {
headings.push({
name: headingMatch[1].trim(),
headStart: headingMatch.index,
contentStart: headingMatch.index + headingMatch[0].length,
});
}
const unfilled = [];
for (let i = 0; i < headings.length; i++) {
const end = i + 1 < headings.length ? headings[i + 1].headStart : body.length;
if (body.slice(headings[i].contentStart, end).includes(PLACEHOLDER)) {
unfilled.push(headings[i].name);
}
}
if (unfilled.length > 0) {
failures.push(
`**Unfilled dropdowns** — please choose a value; these sections still show ` +
`the \`${PLACEHOLDER}\` placeholder: ${unfilled.join(', ')}.`,
);
}
// ── Labels ──────────────────────────────────────────────────────────────── // ── Labels ────────────────────────────────────────────────────────────────
// These labels are expected to already exist in the repo — managing the // These labels are expected to already exist in the repo — managing the
// repo's label set is the maintainer's job, not this workflow's. We check a // repo's label set is the maintainer's job, not this workflow's. We check a

View File

@@ -32,7 +32,7 @@ module.exports = async ({ github, context, core }) => {
// keyword + #NNN, or a full issue URL (e.g. .../issues/123) — the strict // keyword + #NNN, or a full issue URL (e.g. .../issues/123) — the strict
// keyword-prefixed form previously false-flagged correctly-linked PRs. // keyword-prefixed form previously false-flagged correctly-linked PRs.
const linkedSection = section('Linked Issue'); const linkedSection = section('Linked Issue');
const hasIssueRef = /#\d+/.test(linkedSection) || /\/issues\/\d+/.test(linkedSection); const hasIssueRef = /#\d+\b/.test(linkedSection) || /\/issues\/\d+/.test(linkedSection);
if (!linkedSection || !hasIssueRef) { if (!linkedSection || !hasIssueRef) {
problems.push('**Linked Issue** — add a reference like `Fixes #NNN`, a bare `#NNN`, or a link to the issue.'); problems.push('**Linked Issue** — add a reference like `Fixes #NNN`, a bare `#NNN`, or a link to the issue.');
} }
@@ -48,10 +48,12 @@ module.exports = async ({ github, context, core }) => {
problems.push('**Checklist** — check the duplicate-search box to confirm you searched existing issues and PRs.'); problems.push('**Checklist** — check the duplicate-search box to confirm you searched existing issues and PRs.');
} }
// 5. How to Test must have at least one numbered step. // 5. How to Test must contain enough real detail for a reviewer to act on.
// Any format is fine — numbered steps, prose, the commands you ran, or a
// code block — so we only require non-trivial content, not a specific shape.
const howTo = section('How to Test'); const howTo = section('How to Test');
if (!howTo || !/\d+\.\s*\S/.test(howTo)) { if (howTo.length < 30) {
problems.push('**How to Test** — add at least one numbered step a reviewer can follow to verify this works.'); problems.push('**How to Test** — explain how a reviewer can verify this change. Numbered steps, the commands you ran, or a short code block all work — give a sentence or two of real detail (not just "tested locally").');
} }
// ── Comment ────────────────────────────────────────────────────────────── // ── Comment ──────────────────────────────────────────────────────────────