GCP

This page describes steps to follow to setup real-time visualization of GCP Assets (vpc, subnet and virtual server).

Real-time visualization is only supported for vpc, subnet and virtual server resource types as of today (Feb 08, 2024).

Changelog

Feb 8, 2024

Created initial version descripting steps for setting up real-time visualization of GCP assets (vpc, subnet and vm).

Feb 10, 2024

Updated instructions.

Steps for setting up real-time updates

1. Create a topic

Go to console -> topics (use search bar to jump to this).
Add a topic with mc-mgmt-events id (see below screenshot for other details)

Click on newly created topic in the list of topics and then click on TRIGGER CLOUD FUNCTION link.
Create a function with details as follows -
- Basics
  - Environment - 2nd gen
  - Function name - process-mc-events
  - Region - Choose a region of your choice
- Trigger - Leave it unchanged ( make sure it is having topic created in the previous step selected under Cloud Pub/Sub topic field.
- Runtime, build, connections and security settings (Choose other settings as per your convenience)
  - Runtime
    Timeout - 30 seconds
    Runtime environment variables -
    TEAM_ID - d65e32d2-e18f-49a7-8cbf-e19205772ea0
    CLOUD_ACCOUNT_ID - You can get this from MechCloud console -> Infrastructure -> Cloud Accounts page.
    ACCESS_TOKEN - Once you are logged into MechCloud, simply open a new tab, and enter https://portal-preview.mechcloud.io/oauth2/auth1 url in the address bar. It will print your access/jwt token.
  - Connections
    Ingress Settings - Allow internal traffic only
- Source code
  - Runtime - Python 3.12
  - Entry point - process_event
  - Source code - Inline editor
- Make sure requirements.txt has following additional modules -
  - requests==2.31.0
- Add following code under main.py file-

import functions_framework, json, logging, base64, os, requests

@functions_framework.cloud_event
def process_event(cloud_event):

    try:
        print(cloud_event)

        event = base64.b64decode(cloud_event.data["message"]["data"])
        event_str = event.decode('utf-8')
        print('Event : ' + event_str)

        event_json = json.loads(event)
        
        service_name = event_json['protoPayload']['serviceName']
        method_name = event_json['protoPayload']['methodName']
        
        print("Service : '{}', Method : '{}'".format(service_name, method_name))

        team_id = os.environ.get('TEAM_ID')
        if not team_id:
            raise ValueError("TEAM_ID environment variable is missing")
        cloud_account_id = os.environ.get('CLOUD_ACCOUNT_ID')
        if not cloud_account_id:
            raise ValueError("ACCOUNT_ID environment variable is missing")
        access_token = os.environ.get('ACCESS_TOKEN')
        if not access_token:
            raise ValueError("ACCESS_TOKEN environment variable is missing")

        endpoint_url = f"https://mechcloud-preview-asia.mechcloud.io/mechcloud-turbine-discovery/v1.0/sync/events?cloudAccountId={cloud_account_id}"

        headers = {
            'Authorization': f"Bearer {access_token}",
            'Mc-Team-Id': f"{team_id}",
            'Referer': 'https://portal-preview.mechcloud.io/',
            'Content-Type': 'application/json'
        }

        response = requests.post(
                        endpoint_url, 
                        headers=headers, 
                        timeout=(5, 30), 
                        data=event_str
                    )

        status_code = response.status_code
        print('Status code : ' + str(status_code))
        print('Response : ' + response.text)
        if status_code == 200: 
            return f"OK"
        else:
            return f"Not OK"    
    except Exception as ex:
        print('Exception occured.')
        logging.error(ex, exc_info=True)
        
        return f"Not OK"

2. Create log routing sinks

To visualize GCP resource in real-time, we will need to create one log routing sink per resource type (e.g. vpc). You can add a log routing sink by going to console -> logging (use search bar to jump to this) -> Log router and click on Create sink link to add a sink. Make sure you have following log routing sinks created -

PreviousCloudTrail NextRouting

Last updated 8 months ago

import functions_framework, json, logging, base64, os, requests @functions_framework.cloud_event def process_event(cloud_event): try: print(cloud_event) event = base64.b64decode(cloud_event.data["message"]["data"]) event_str = event.decode('utf-8') print('Event : ' + event_str) event_json = json.loads(event) service_name = event_json['protoPayload']['serviceName'] method_name = event_json['protoPayload']['methodName'] print("Service : '{}', Method : '{}'".format(service_name, method_name)) team_id = os.environ.get('TEAM_ID') if not team_id: raise ValueError("TEAM_ID environment variable is missing") cloud_account_id = os.environ.get('CLOUD_ACCOUNT_ID') if not cloud_account_id: raise ValueError("ACCOUNT_ID environment variable is missing") access_token = os.environ.get('ACCESS_TOKEN') if not access_token: raise ValueError("ACCESS_TOKEN environment variable is missing") endpoint_url = f"https://mechcloud-preview-asia.mechcloud.io/mechcloud-turbine-discovery/v1.0/sync/events?cloudAccountId={cloud_account_id}" headers = { 'Authorization': f"Bearer {access_token}", 'Mc-Team-Id': f"{team_id}", 'Referer': 'https://portal-preview.mechcloud.io/', 'Content-Type': 'application/json' } response = requests.post( endpoint_url, headers=headers, timeout=(5, 30), data=event_str ) status_code = response.status_code print('Status code : ' + str(status_code)) print('Response : ' + response.text) if status_code == 200: return f"OK" else: return f"Not OK" except Exception as ex: print('Exception occured.') logging.error(ex, exc_info=True) return f"Not OK"

Sink Name

Sink Destination

Logs to include in sink