Retrieving GCP Billing Data

You can configure vCommander to retrieve Google Cloud Platform billing data. Retrieving billing data improves the accuracy of the Cloud Billing Report and Cost Analytics because these reports include metered usage costs such as Storage and Network I/O. Any credits in the billing data are also accounted for, and the service costs are automatically adjusted for those credits.

This topic assumes you understand the basics of Google Cloud billing in general, including exporting billing data to BigQuery.

In this topic:

About billing data retrieval

Once you've configured billing data retrieval as explained in this topic, vCommander retrieves a maximum of 60 days' worth of billing data. vCommander can retrieve data from before the GCP account was added as a managed system. New billing data is then retrieved nightly.

Unlike AWS and Azure, GCP billing data doesn't include VM-level costs. VM billing records and the VM Billing Report use estimated costs, rather than historical costs for GCP.

vCommander also retrieves billing data for services that have never been managed by vCommander. For example, if a GCP account was added as a managed system 90 days ago, and you configure vCommander to retrieve billing data today, vCommander retrieves the last 60 days' worth of billing data for all services that have existed at any point over the last 60 days. As another example, if you add a GCP account as a managed system today, and you configure vCommander to retrieve billing data today, vCommander retrieves the last 60 days' worth of billing data for all services that have existed at any point over the last 60 days, including services that were never managed by vCommander.

GCP starts exporting billing data once export has been configured in the GCP Console, and doesn't backfill billing data. This means that vCommander can only retrieve billing data from the time that billing data export is configured in GCP Console.

Billing data also includes costs for resources and services that aren't managed by vCommander but will show up on your bill, such as Cloud Datastore, BigQuery, Stackdriver Logging and Cloud CDN. When a resource not managed by vCommander is discovered in the billing data, ownership is assigned to all costs, based on the Default Ownership policy and the service's location in the Operational or Deployed tree, for the purposes of Cost Analytics in both vCommander and the Service Portal.

vCommander displays all public cloud costs in US dollars (USD).

How the GCP billing model maps to vCommander costing

A GCP managed system may include multiple projects, so a single managed system may be associated with multiple billing accounts.

Likewise, if you've added multiple GCP managed systems, the same project may be included in multiple managed systems, meaning that multiple managed systems may be linked to the same billing account.

To help you understand this relationship, consider the following diagram:

Relationship Between GCP Billing Data and vCommander Managed Systems

Note the following about this example:

  • When you export a BigQuery dataset, you must select its project location. In this example, BigQuery Dataset 1 is located in Project 1, Big Query Dataset 2 is located in Project 3, and BigQuery Dataset 3 is located in Project 7.
  • When you configure billing data retrieval for Managed System 1, vCommander retrieves billing data from BigQuery Dataset 1 and 2. Because Project 1 is visible to both managed systems, billing data is automatically retrieved for this project in Managed System 2, even if you haven't configured billing data retrieval for Managed System 2.
  • Because Managed Systems 1 and 2 share a billing account, once billing data is retrieved for Billing Account 1, charges for Project 1, 2 and 6 are added to both managed systems.
  • If you want to retrieve billing data for Project 7 and 8, you must also configure billing data retrieval for Managed System 2, pointing to BigQuery Dataset 3.
  • Billing is not enabled for Project 5.

Overview of steps

To retrieve GCP billing data, there are several mandatory steps:

  1. Export billing data to BigQuery.
  2. Enable the Cloud Billing API.
  3. Grant dataset permissions to the vCommander service account.
  4. Configure the Default Ownership policy.
  5. Retrieve GCP billing data.

Export billing data to BigQuery

To be able to retrieve GCP billing data in vCommander, you must export billing data to Google BigQuery.

To learn how, see Export Billing Data to BigQuery in the Google Cloud documentation.

Notes:

  • vCommander supports only v1 of the BigQuery data export tables; the original version is not supported.
  • vCommander doesn't support GCP billing data exported to CSV or JSON. Only BigQuery datasets are supported.

Enable the Cloud Billing API

In the GCP Console, ensure that the Cloud Billing API is enabled for the project where the vCommander service account is located.

To learn how to enable APIs for a project, see Enabling and Disabling Services in the Google Cloud documentation.

Grant dataset permissions to the vCommander service account

Ensure that the service account has access to the exported BigQuery dataset. See Grant permissions to the vCommander service account to learn more.

Configure the Default Ownership policy

GCP charges aggregated CPU and memory usage costs, grouped by project and service type. GCP billing data doesn't include costs for individual VMs. Before you retrieve billing data, you must configure the Default Ownership policy, to ensure that cost analytics are accurate, and costs are assigned to the proper owners and organizations.

For example, if you configure the Default Ownership policy to assign ownership of all services in GCP Project X to vCommander Organization A, vCommander can then assign the total cost of all services in Project X to vCommander Organization A.

If you don't configure the Default Ownership policy before retrieving billing data, ownership won't be applied to the retrieved costs. This means that:

  • Cost Analytics information won't be accurate
  • in the Service Portal, charges won't be visible
  • in vCommander, no ownership information will be assigned to the charges

While the Default Ownership policy applies ownership only to new services, vCommander also uses this policy to assign ownership to newly retrieved costs for the purposes of Cost Analytics.

To learn how to configure the Default Ownership policy, see Assigning Service Ownership with Default Ownership Policies.

Retrieve GCP billing data

Access through:

Views menu > Operational or Deployed

Available to:

vCommander Roles of Superuser and Enterprise Admin; Administrator Access Rights

You need to configure billing data retrieval in vCommander at the managed system level.

If you have multiple GCP managed systems and they use the same single billing account, billing data for all of the managed systems will be retrieved at the same time. There's no need to configure billing data retrieval for the other managed systems.

Once billing data retrieval is configured for GCP, estimated costs are replaced with billing data costs in Cost Analytics. Cost analytics won't include costs for services associated with any billing accounts whose data hasn't been retrieved, so we recommend retrieving billing data for all associated billing accounts.

  1. Right-click a GCP managed system in the tree and select Retrieve Billing Data.

    The Billing Data Settings dialog appears. The first column lists all billing accounts associated with this managed system. The second column lists all projects associated with each billing account, whether or not the project is part of this managed system.

    If you see the message "To retrieve billing data for this managed system, you must enable the Cloud Billing API", see Enable the required APIs.

  2. For Project Containing BigQuery Dataset, select the project where your BigQuery dataset is stored for this billing account.
  3. Enter the Dataset Name.

    You can find the dataset name in the GCP Console. In the following example, the BigQuery dataset is located in Embotics Project X, and the dataset name is embotics_main_billing_export.

    Here's what this example looks like in vCommander:

  4. Click Test to verify that the dataset exists, and that this managed system has access to it.
  5. If you see the warning "Dataset found, but no billing data can be found in the dataset", this means that no billing data is available in the specified dataset. If you've recently exported billing data to this BigQuery dataset, you can ignore this warning, because it might take a few hours for data to be available from GCP. vCommander will retrieve billing data once it's available.
  6. If you see the error "The dataset 'x' cannot be found. Verify the dataset name and ensure that the service account used to add the selected project's managed system has access to the dataset", verify that:
    • the dataset is located in the selected project
    • you have selected the project in the correct managed system. If the project is visible to multiple managed systems, but the dataset is not accessible to all managed systems, you must select the project under the managed system that has access to the dataset. In the image below, "Embotics Project X" is visible to two managed systems, but the embotics_main_billing_export dataset is accessible only to the managed system named GCP - Embotics Automation, so "Embotics Project X" must be selected under GCP - Embotics Automation.

  7. Repeat steps 2-4 for each billing account that contains data you want to retrieve.
  8. Click OK to close the dialog and retrieve billing data.

    If you see an error message, to close the dialog, you must either click Not Configured and OK, or click Cancel.

Disabling billing data retrieval

Access through:

Views menu > Operational or Deployed

Available to:

vCommander Roles of Superuser and Enterprise Admin; Administrator Access Rights

To disable billing data retrieval for a GCP managed system:

  1. In the tree, right-click the GCP managed system for which you want to stop retrieving billing data and select Retrieve GCP Billing Data.
  2. In the GCP Billing Data Settings dialog, for each billing account that you want to disable, select Not Configured and click OK.

    vCommander will no longer retrieve billing data for this billing account.

If other GCP managed systems share this billing account, billing data retrieval will also be disabled for those managed systems.