phantom

Overview ¶

phantom finds ruby/spec examples that can pass BECAUSE A METHOD IS MISSING.

Why it exists (#850). A spec that asserts an exception class cannot tell why it got one. core/array/rfind_spec.rb has

it "raises a NoMethodError if the ifnone argument does not respond to #call …" do
  -> { [1, 2, 3].rfind(42) { |x| false } }.should.raise(NoMethodError)
end

which means "42 is not callable". On a build where `Array#rfind` does not exist at all, `[1,2,3].rfind(42)` raises NoMethodError for THAT reason and the example passes. Measured on 3a3fb818, that file scored `pass=1 error=15` with no rfind in the interpreter: fifteen honest errors and one pass the gate counted for a method it did not have.

So BASELINE can hold passes that are not conformance, and nothing distinguished them. This names them.

WHAT IT DOES. For every `describe "X#y"` or `describe "X.y"` block containing a NoMethodError expectation, it asks a real rbgo whether that method exists — `X.method_defined?(:y)` (or `private_method_defined?`, since a module_function is private) for the `#` form, `X.respond_to?(:y)` for the `.` form. No instance is constructed, so a receiver whose `new` needs arguments is still answerable. An answer of "no" means that example's pass, if it passes, is suspect.

⛔ WHAT IT CANNOT SEE, printed on every run rather than left in this comment: it reads the describe string, so an example whose receiver is not what the describe names is outside its population, and a describe naming an operator method (`[]`, `<=>`, `+`) is skipped because those do not match the identifier shape. Same class of limit as cmd/guards declaring its example counts a lower bound. A census that does not state its population is a census that will be read as a total.

Exit status: 0 nothing suspect, 1 suspects found, 2 refused — could not read the corpus, could not run the interpreter, or its own control failed.