Golang proposal: testing: allow examples the use of testing.T/B/F for test framework examples

Proposal Details

Today, examples are tests presented alongside godoc as a "func main()" typically, as shown in https://go.dev/blog/examples, and is executable like that.

I'm trying to write examples for a test framework I created (https://github.com/hugelgupf/vmtest) that requires the use of testing.TB. To create an executable example in pkg.go.dev, examples have to be able to take testing.T or testing.B as an argument.

IMO, the existing Example "API" should remain as is. If an example is given a testing.T/B/F argument, though, it should be assumed to require it and be presented that way in the documentation as well, e.g.

func ExampleFunctionname(t *testing.T) {
  t.Logf("foobar")
  // Output: foobar
}

Should appear alongside Functionname in documentation as an executable snippet like this:

package main_test

import (
  "testing"

  "packagepath"
)

func TestExample(t *testing.T) {
    t.Logf("foobar")
}

Comment From: earthboundkid

I wrote a test framework and had this same problem. I usually use example driven development, but it was sort of hard to do that for a test framework. I had to make a dummy testing.TB, and it didn't always work very well. For example, testing.TB doesn't have T.Run, so I just stub that with code that compiles but doesn't execute.

Comment From: ianlancetaylor

This seems fairly special purpose. Examples don't have to have outputs. Should we also permit example functions to take a slog.Logger argument? It seems hard to know when to stop.

Comment From: earthboundkid

I think the proposal should be instead to make it easier to make a testing.T/B/F for test purposes. So the code would look like

func ExampleWhatever() {
    result := testing.RunTest(func(t *testing.T) {
        // do stuff with *testing.T
    })
    fmt.Println("Test failed was", result.Failed, "because", result.Output)
    // Output: 
    // Test failed was true because error: bad potatoes
}

This could also be used in a TestFunction to test a testing framework more easily.

Comment From: dolmen

In my proposal #74009 I have pointed that the Go Playground already allows to run Test...(*testing.T) functions, so the execution backend for x/pkgsite is already available. An Example...(*testing.T) would just have to be renamed to Test...

I think that an Example...(*testing.T) should NOT be executed like existing Example...() functions that specify the expected output, but instead like a standard test function that uses t.Fail/Error/Fatal/...

I don't see much value in supporting benchmarks (*testing.B) or fuzzing (*testing.F) in examples. If there is a need there, that would be very niche cases. Also, they are not supported by the Go Playground.

Comment From: dolmen

Playable examples are exposed in stdlib by package go/doc.

Comment From: dolmen

In [CL 679655](https://go.dev/cl/679655] I have implemented the go/doc changes that allow to expose those Example...(*testing.T) examples in pkgsite without any changes to pkgsite itself (just rebuild pkgsite with the modified go toolchain). In that proposal the example tests are standard test functions and there is no special processing of // Output: comments.

I haven't yet implemented changes to package testing that would add those Example...(*testing.T) tests in the default list of tests.

I'm waiting for feedback. Maybe this partial implementation could help this proposal (#74009) to be considered for review?