Measure runtime code coverage caused by running Postman/newman tests

[anishkny]

First off great job on Deno so far - kudos to the team! 🎉🎉🎉

I would like to measure code coverage of a Deno program (an HTTP server) exercised by running Postman tests against it (via the newman CLI runner).

In Node.js I would achieve this by running nyc node server.js, and then running my Postman tests against the server that was started.

How can I do something similiar in Deno? Maybe something like this:

deno run --coverage ./main.ts &
newman ...          # Run Postman tests using newman CLI
kill $SERVER_PID    # Kill deno process to dump coverage numbers

Or, even better, everything in one command to start server, run external tests, and end server:

deno run --coverage-command 'newman ...' ./main.ts

ETA: Looks like there is a workaround as detailed by @zachauten below. I have spinned up a repo demonstrating this at anishkny/deno-api-tests-coverage.

ETA2: Opened #16440 for this.

[zachauten]

Coverage is generated with the --coverage flag on the test command. You pretty much have the right idea, but you'll need to start the server inside a test. If you're using oak, your test could look something like:

1. import server to test
2. create an abort controller and start the server with it.
3. shell our to run the tests.
4. use the abort controller to kill the server.
5. run the server with deno tests --coverage

Example (this comes from some boilerplate i wrote a while ago, it may need to be updated to current deno/oak api's):

import { server } from "./server.ts";

Deno.test("test server", async t => {

  const application = app();
  const controller = new AbortController();
  const { signal } = controller;
  const listen_promise = application.listen({ port: 8080, signal });

  // create subprocess to run tests with "newman"
  const cmd = ["newman", "..."];
  const p = Deno.run({ cmd });
  const { success } =  await p.status();
  assertEquals(success, true);

  controller.abort();
  await listen_promise;
});

[anishkny]

Thanks for the suggestion. I'll try it out but it seems roundabout.

It will be way more intuitive if we get something similar to deno run --coverage ./main.ts.

[gm112]

I think you mean to run v8's profiler tool. The code coverage data is more for getting a read on what code is covered by the given unit tests, if that makes sense? v8's profiler tool will highlight time spent in every method, which probably is what you want... See the following v8 flags:

    --prof
    --log-internal-timer-events
    --log-timer-events
    --track-gc
    --log-source-code
    --track-gc-object-stats

[anishkny]

With respect @gm112 I know what I need. The suggested answer above provides exactly what I need (which lines/decisions were covered in a running Deno process, as a result of another test process interacting with it) but is unwieldy.

My life would be much simpler if this existed:

deno run --coverage ./main.ts    # I need this!