bigquery: add simple benchmarks

[pongad]

It runs a series of queries and report

1. How long until we get the first result back
2. How long we take to iterate through all results

[dwsupplee]

Very nice, thanks @pongad! We have a similar benchmarking test for logging located at: https://github.com/GoogleCloudPlatform/google-cloud-php/tree/master/tests/perf. WDYT about having this benchmark live there as well?

[pongad]

@dwsupplee Sounds good, thank you for letting me know

[pongad]

@dwsupplee PTAL

[jdpedrie]

Hi @pongad,

I took the liberty of modifying your test file to work as a PHPUnit test case (the test runner we use for all our test suites). I also modified things a bit to work with the latest changes in master (from our changes to BigQuery introduced in #686). Please take a look and tell me what you think.

/tests/perf/BigQueryTest.php

<?php
/**
 * Copyright 2017 Google Inc.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

namespace Google\Cloud\Tests\Perf;

use Google\Cloud\BigQuery\BigQueryClient;

/**
 * @group bigquery
 */
class BigQueryPerfTest extends \PHPUnit_Framework_TestCase
{
    const SOURCE = 'queries.json';

    private $client;

    public function setUp()
    {
        $keyFilePath = getenv('GOOGLE_CLOUD_PHP_TESTS_KEY_PATH');
        $this->client = new BigQueryClient([
            'keyFilePath' => $keyFilePath
        ]);
    }

    /**
     * @dataProvider queries
     * @runInSeparateProcess
     */
    public function testPerf($query)
    {
        $start = microtime(true);
        $config = $this->client->query($query);
        $queryResults = $this->client->runQuery($config);

        $rows = 0;
        $cols = 0;
        $firstByteDur = 0;

        foreach ($queryResults as $row) {
            $rows++;
            if ($cols == 0) {
                $firstByteDur = microtime(true) - $start;
                $cols = count($row);
            } elseif ($cols != count($row)) {
                throw new Exception("expected $cols cols, found " . count($row));
            }
        }

        $totalDur = microtime(true)-$start;
        echo "query $query: $rows rows, $cols cols, first byte $firstByteDur, total $totalDur" . PHP_EOL;
    }

    public function queries()
    {
        $queries = json_decode(file_get_contents(__DIR__ .'/'. self::SOURCE), true);
        foreach ($queries as $key => $q) {
            $queries[$key] = [$q];
        }

        return $queries;
    }
}

To run:

vendor/bin/phpunit -c phpunit-perf.xml.dist --group=bigquery

[jdpedrie]

note: to get this working, I had to bump up the PHP process memory limit by modifying phpunit-perf.xml.dist:

<?xml version="1.0" encoding="UTF-8"?>
<phpunit bootstrap="./tests/perf/bootstrap.php" colors="true">
  <testsuites>
    <testsuite>
      <directory>tests/perf</directory>
    </testsuite>
  </testsuites>
  <filter>
    <whitelist>
      <directory suffix=".php">src</directory>
      <exclude>
        <directory suffix=".php">src/*/V[!a-zA-Z]*</directory>
      </exclude>
    </whitelist>
  </filter>
  <php>
    <ini name="memory_limit" value="2048M"/>
  </php>
</phpunit>

[pongad]

@jdpedrie Thank you for picking up the work! A few comments.

1. Is the main motivation of integrating with PHPUnit to run on CI? These tests are quite network and CPU intensive, so I don't think we can get a good reading running in CI. To test on schedule, I think we'd need a dedicated machine and run the script periodically. I'm not completely against the change since it'd still be possible to run this as a test on CLI, but the work might not be necessary.

2. Can we keep queries.json? We hope to eventually create a tool to sync "resource files" like this across google-cloud-{lang} repositories. If we write the queries themselves into a PHP file, the sync tool won't work.

[jdpedrie]

@pongad,

1. Not necessarily run in CI, though as you say, that could be an option in the future. I did this originally because the Logging perf test already uses PHPUnit. As I think more though, I like the idea of keeping uniformity with the other test suites where possible. Using PHPUnit for perf lets us group tests, run them the same way we do the other suites, etc. It also offers some more tangible benefits, such as simple process forking (shown in my example @runInSeparateProcess) to make memory limits less of an issue.
2. Sure! I've updated my example above.

[pongad]

@jdpedrie (1) sounds good to me. Sounds like you already made the changes and have this running right? Would you like to take over this PR?

[jdpedrie]

Sure thing. @pongad, I've opened #708, so I'm going to close this PR.