AWS

Changelog

Feb 14, 2024

• Added following sections -

  • Choosing between Tags and CloudTrail

  • Configuring sync mechanism for a cloud account

  • Steps

• Updated following sections -

  • Details

Real-time visualization is only supported for vpc, subnet and virtual server resource types as of today (Feb 08, 2024).

Details

There are two options available to visualize AWS resources in real-time -

• First option is based on Tags. This option is already supported as of today (Feb 08, 2024).

• Second option is based on CloudTrail. This option is also supported as of today (Feb 14, 2024).

Choosing between Tags and CloudTrail

• CloudTrail

  • This is recommended for new / greenfield resources provided you are comfortable with following cons -

    • It takes some time before an event appears in event history and so this process is slower compared to tags-based approach for real-time visualization.

    • A trail does not provide any advanced filters to excludes all the events other than the one you are interested in. So, it will be capturing a large number of events in the S3 bucket associated with the trail where many of such events will not be processed by the eventbridge rule required for real-time visualization.

  • To consume cloudtrail events using eventbridge, you are required to create at least one trail. However, if you are having more than one trails then you can't select only one of such trails while creating an eventbridge rule which will forward cloudtrail events to an SQS queue. This could be confusing.

• Tags

  • This option is recommended for existing / brownfield resources and in fact, it is the only option which will work for existing / brownfield resources. However, it can be used for newly created resources as well.

  • This option requires that you add a tag called Mc-Asset-Name to all the resources either at the time of creation or post creation so that it can be discovered in MechCloud in real-time.

Configure sync mechanism for a cloud account

• Navigate to MechCloud console -> Infrastructure -> Cloud Accounts page.

• Add an account or update an existing account and choose desired sync mechanism as shown below -

Steps

Both the mechanism requires to define a lambda function and a FIFO queue. Steps for setting up these are described below. Sync mechanism specific steps are described under respective page.

1. Create a Lambda function

Create a python lambda function with the following details -

• Name - push-mgmt-events-to-mechcloud

• Runtime - Python 3.12

• Code

import json
import requests
import os

def lambda_handler(event, context):
    print(event)
    for record in event['Records']:
        message_body = json.loads(record['body'])
        
        team_id = os.environ.get('TEAM_ID')
        if not team_id:
            raise ValueError("TEAM_ID environment variable is missing")
        cloud_account_id = os.environ.get('CLOUD_ACCOUNT_ID')
        if not cloud_account_id:
            raise ValueError("ACCOUNT_ID environment variable is missing")
        access_token = os.environ.get('ACCESS_TOKEN')
        if not access_token:
            raise ValueError("ACCESS_TOKEN environment variable is missing")

        endpoint_url = f"https://mechcloud-preview-asia.mechcloud.io/mechcloud-turbine-discovery/v1.0/sync/events?cloudAccountId={cloud_account_id}"

        headers = {
            'Authorization': f"Bearer {access_token}",
            'Mc-Team-Id': f"{team_id}",
            'Referer': 'https://portal-preview.mechcloud.io/',
            'Content-Type': 'application/json'
        }

        try:
            response = requests.post(
                        endpoint_url, 
                        headers=headers, 
                        timeout=(5, 30), 
                        data=json.dumps(message_body)
                    )
            response.raise_for_status()
            print(f"Message pushed successfully: {response.text}")
        except requests.exceptions.RequestException as e:
            print(f"Error pushing message: {e}")

• Make sure you have following environment variables defined -

  • TEAM_ID - This is the id generated by MechCloud for a team which a cloud account belongs to. Its value is d65e32d2-e18f-49a7-8cbf-e19205772ea0 for the default team.

  • CLOUD_ACCOUNT_ID - This is the unique id which is generated by MechCloud when a cloud account is registered. You can get this from MechCloud console -> Infrastructure -> Cloud Accounts page.

  • ACCESS_TOKEN - Once you are logged into MechCloud, simply open a new tab, and enter https://portal-preview.mechcloud.io/oauth2/auth1 url in the address bar. It will print your access token.

• Update timeout to 30 seconds under Configuration -> General Configuration -> Timeout.

2. Create a FIFO queue

Create a fifo queue with following details -

• Name - management-events.fifo

• Type - FIFO

• Make sure Content-based deduplication is enabled.

• You can adjust other settings as per your need.

• Configure the lambda function created in previous step under Lambda triggers of this queue.

3. Create an Event Bridge rule

Create an event bridge rule with the following details -

• Define rule detail

  • Name - mc-instance-state-change-events

  • Event bus - default

  • Make sure that Enable the rule on the selected event bus is enabled.

  • Select Rule with an event pattern under Rule type.

• Build event pattern

  • Creation method - Custom pattern (JSON editor)

  • Event pattern

{
  "source": ["aws.ec2"],
  "detail-type": ["EC2 Instance State-change Notification"]
}

• Select target(s)

  • Target 1

    • Target types - AWS service

    • Select a target - SQS Queue

    • Queue - management-events.fifo

    • Message group ID - mc-instance-state-change-events