mapping-gen

Overview ¶

Package main (former2.go): the second of issue #52's two authoritative sources, iann0036/former2 - a community-maintained CloudFormation generator that, per resource type it supports, already carries both the CFN type it emits and the Terraform type the same live object would import as. That per-resource pairing is exactly a mapping-gen row, from someone else's independent read of both APIs.

former2's per-resource database is not the single js/mappings.js file issue #52 names - that file (checked directly against the pinned commit) is render-time helper code with no CFN/TF pairs in it at all. The pairs live one JS object literal per resource, spread across the ~150 files under js/services/*.js, each pushed onto former2's own tracked_resources array in the shape:

tracked_resources.push({
    ...
    'type': 'AWS::EC2::Instance',
    'terraformType': 'aws_instance',
    ...
});

extractFormer2Rows reads that shape tolerantly rather than evaluating the JavaScript: former2 is community data (issue #52's own framing - "a wrong entry maps a wrong resource"), and a regex over a well-worn literal pattern that degrades to "found nothing" on a shape it does not recognize is a safer failure mode for an unreviewed upstream than a JS engine embedding would be.

Content pin and acquisition for former2's js/services/*.js (issue #52), the #42 pattern again: {digest, accepted, roster-ish summary}. former2 is fetched at a pinned commit sha (the tag concept doesn't apply to a repo that does not cut releases); the source is the whole js/services directory's concatenated text, fetched as a tarball of the pinned commit rather than ~150 individual file requests.

mapping-gen generates live/mapping.json, the TF-to-CFN type mapping artifact everything in #40 flows through (issue #43).

It joins two committed rosters - live/survey-full.json's 1,691 Terraform AWS resource types (issue #41) and live/registry.json's 1,653 CloudFormation Registry types (issue #42) - against the curated overlay at tools/mapping-gen/overlay.json plus issue #52's two generated/sourced tables (names_data.hcl's service-alias join and former2's per-resource pairing, each committed as its own artifact - namesdata-generated.json, former2-rows.json - so a default run needs no network, same as every other artifact this tool reads), and writes one row per TF type:

{"tf_type": "aws_s3_bucket", "cfn_type": "AWS::S3::Bucket", "via": "name", "fold_parent": null, "note": null}

via is name (the heuristic in heuristic.go derived the CFN type from the TF type's own name), alias (the overlay asserts the pair by hand), service-alias (servicealias.go's heuristic v2 matched the TF type's resource tokens against one CFN service's own resource names, the service now coming from names_data.hcl's generated table, the overlay's service_aliases kept only for prefixes the source disagrees with or does not cover), former2 (iann0036/former2's own independent per-resource CFN/TF pairing, for a TF type none of the above could resolve), fold (the TF type is a property-child of a CFN parent - fold_parent carries the parent's CFN type and cfn_type stays null), or one of issue #53's three terminal-taxonomy values for a TF type with no CFN mapping and no fold parent: tf-only (a provider-side construct with no cloud resource of its own - taxonomy.go's mechanical classifier, or a curated overlay.json tf_only entry), cfn-unmodeled (a real resource CFN does not model - curated only, overlay.json's cfn_unmodeled table), or deprecated-service (the TF type's prefix is in live/residue.go's DeprecatedServices, and every registry type in that CFN service ships no working handler - taxonomy.go's mechanical classifier). Anything left after all of that is none, issue #53's own named "unclassified" remainder - note says why when known, and the header's counts.unclassified is exactly how many rows are still here.

Usage, from anywhere in the checkout:

go run ./tools/mapping-gen

-refresh-names-data and -refresh-former2 re-fetch (network, or an explicit -names-data-hcl/-former2-src override that also seeds the disk cache) and rewrite this tool's two committed source artifacts; each prints the pin block to paste into the matching *_source.go if the fetched content's digest has moved (or, with the matching accept env set, proceeds and prints the new block anyway).

Package main (namesdata.go): the first of issue #52's two authoritative sources for the service-alias table that overlay.json's service_aliases section used to hand-curate.

hashicorp/terraform-provider-aws ships names/data/names_data.hcl: the provider's own reference data for every AWS service it wires, one `service "prefix" { ... }` block per family, each carrying (among other things) the AWS SDK's own service identifier (sdk.id, e.g. "EC2") and, for a family the provider itself further splits (ec2's nine sub_service blocks - vpc, ipam, transitgateway, ...), a nested `sub_service "prefix" { ... }` block per split.

The join this file computes is exactly the one issue #52 names: TF prefix -> AWS service id -> CFN service name. A block's own name is always its resource prefix (resource_prefix.correct is, in every instance checked against the pinned v6.58.0 file, precisely "aws_" plus the block's own label plus "_" - the field carries no information the label itself does not already carry, so the label is used directly and the correct field is not parsed at all).

Every one of the file's 9 sub_service blocks (all under "ec2") carries an empty sdk.id of its own - the file's own signal that the split is a TF naming convenience, not a distinct AWS API surface, so a sub_service's effective AWS id is always its parent's sdk.id (or the parent's provider_name_upper when even that is empty), never anything of the sub_service block's own. vpc -> EC2 falls out of this rule directly: vpc is a sub_service of ec2, so its effective id is ec2's own "EC2".

The AWS id (sdk.id when non-empty, else names.provider_name_upper) is matched against live/registry.json's CFN service names case- and separator-insensitively (sdk.id often carries spaces the CFN name never does, e.g. "Application Auto Scaling" -> ApplicationAutoScaling): a unique match generates a service_aliases-shaped entry, prefix -> that CFN service; no match, or more than one CFN service sharing the same normalized spelling (never observed against the pinned registry, but checked rather than assumed), is a genuine wrinkle - most of them services CloudFormation simply does not model as their own resource family, e.g. "cognitoidp" (Cognito's IDP-only SDK client, folded under CloudFormation's own "Cognito" service, itself reached through the "cognito" block, not this one) - counted and reported rather than silently dropped.

Content pin and acquisition for names_data.hcl (issue #52), following the #42 pattern: {digest, accepted, roster-ish summary}. Unlike the CloudFormation Registry zip (a "latest" artifact with nothing to version), names_data.hcl is fetched at a pinned provider release tag - a real version exists, so the pin's Digest is a belt-and-suspenders check that the tag's content has not moved underneath the tag (rare, but not impossible for a force-pushed tag) rather than the only way to detect drift the CFN zip pin exists for. Any digest mismatch refuses outright; there is no soft-warn tier the way the CFN zip pin has, because there is no legitimate "upstream repackaged the same content" case for a fetch pinned to an immutable tag.

Package main (taxonomy.go): issue #53's two mechanical classifiers, run by buildMapping (mapping.go) over whatever is still via:none with the generic unexplainedNoteText after every mapping source - overlay, both name heuristics, and former2 - has had a turn. Each classifier requires corroboration beyond a name match before it assigns a terminal via; a pattern match with no corroborating evidence is left via:none for a later family sweep to judge by hand (tools/mapping-gen/overlay.json's tf_only or cfn_unmodeled tables), rather than guessed at here. Being conservative is the point: a wrong terminal classification is worse than an honestly counted unclassified row.

There is no general mechanical cfn-unmodeled classifier here, on purpose. Proving a real resource has no CFN model at all - as opposed to simply not having been found by name - is exactly the per-family judgment call issue #53's own workplan defers to the follow-up sweeps; a classifier that mechanically promoted every still-unmatched row to cfn-unmodeled would just be via:none relabeled. via:cfn-unmodeled stays curated-only (overlay.go's CFNUnmodeled table) for that general population, until a sweep supplies the judgment with its evidence.

The deprecated-service branch is a narrow, evidenced exception (issue #246): it already requires a type's TF prefix to sit on live/residue.go's curated DeprecatedServices list AND its family's entire registered CFN footprint to be handler-less (deprecatedServiceEligible) before it says anything at all. Given that much corroboration, a further per-type check - does any real CFN type anywhere in the registry actually name this resource, per the same global name index every other row in this package is joined against - is strong enough to settle cfn-unmodeled for that type specifically, without generalizing to "unmatched implies unmodeled" for the rest of the unclassified population.