4 Replies Latest reply on Sep 24, 2018 10:34 PM by Amrendra Kumar Singh

    AWS - AWS.EC2.DescribeVpcPeeringConnections output parsing

    Amrendra Kumar Singh
      Share This:

      I want to parse the output of “AWS.EC2.DescribeVpcPeeringConnections” API call. Screenshot of DA generated is attached here with name “Discovry.png”.

       

      Looking into this data I can see output is consist of nested “KVPDict” data type. Raw data format is attached here with name “Discovrey2.png”

       

      For example: Value for “AccepterVpcInfo” have nested KVPDict.

       

      How should I parse the nested KVPDict and nested KVPDict to get the key and value?

       

       

      If I use this logic I get output mentioned below where I can see KVPDict type data and some are nested too:

       

      target_groups := discovery.AWS.EC2.DescribeVpcPeeringConnections(da);

      for item in target_groups do

           for key in item do

                if key <> "__index" then

                     val_tal := item[key];

                     log.info("%key%:%val_tal%");

                 end if;

           end for;

      end for;

      Note: String marked in bold are keys.

       

      AWS.EC2.DescribeVpcs (0): 173ms: Status:KVPDict({'Message': 'Active', 'Code': 'active'})

      AWS.EC2.DescribeVpcs (0): 173ms: __type:VpcPeeringConnection

      AWS.EC2.DescribeVpcs (0): 175ms: Tags:[KVPDict({'Key': 'Name', 'Value': 'Classic-ModernPeering'})]

      AWS.EC2.DescribeVpcs (0): 177ms: __identity:pmm-xxxx00000

      AWS.EC2.DescribeVpcs (0): 178ms: RequesterVpcInfo:KVPDict({'PeeringOptions': KVPDict({'AllowEgressFromLocalVpcToRemoteClassicLink': False, 'AllowDnsResolutionFromRemoteVpc': False, 'AllowEgressFromLocalClassicLinkToRemoteVpc': False}), 'VpcId': 'vpc-00000000005', 'Region': 'ap-southeast-1', 'OwnerId': '90000000000, 'CidrBlockSet': [KVPDict({'CidrBlock': 'xx.00.yyy.0/cc'})], 'CidrBlock': 'xx.00.yyy.0/cc'})

      AWS.EC2.DescribeVpcs (0): 179ms: VpcPeeringConnectionId:pcc-1bbbbb88888

      AWS.EC2.DescribeVpcs (0): 179ms: AccepterVpcInfo:KVPDict({'OwnerId': '799999999999999', 'Region': 'ap-southeast-1', 'VpcId': 'vpc-9999999999e', 'CidrBlock': 'xx.00.yyy.0/cc', 'CidrBlockSet': [KVPDict({'CidrBlock': 'xx.00.yyy.0/cc'})]})

      AWS.EC2.DescribeVpcs (0): 180ms: Status:KVPDict({'Message': 'Active', 'Code': 'active'})

      AWS.EC2.DescribeVpcs (0): 181ms: __type:VpcPeeringConnection

      AWS.EC2.DescribeVpcs (0): 181ms: Tags:[KVPDict({'Key': 'Name', 'Value': 'Classic-CloudFoundryPeering'})]

      AWS.EC2.DescribeVpcs (0): 182ms: __identity:pbb-yyyy99999

      AWS.EC2.DescribeVpcs (0): 182ms: RequesterVpcInfo:KVPDict({'PeeringOptions': KVPDict({'AllowEgressFromLocalVpcToRemoteClassicLink': False, 'AllowDnsResolutionFromRemoteVpc': False, 'AllowEgressFromLocalClassicLinkToRemoteVpc': False}), 'VpcId': 'vpc-9nnnn00000', 'Region': 'ap-southeast-1', 'OwnerId': '9888888888', 'CidrBlockSet': [KVPDict({'CidrBlock': 'xx.00.yyy.0/cc'})], 'CidrBlock': 'xx.00.yyy.0/cc'})

      AWS.EC2.DescribeVpcs (0): 183ms: VpcPeeringConnectionId:pff-dppppppppp

      AWS.EC2.DescribeVpcs (0): 184ms: AccepterVpcInfo:KVPDict({'OwnerId': '8222222222222', 'Region': 'ap-southeast-1', 'VpcId': 'vpc-4kkkkkkkkkkkk, 'CidrBlock': 'xx.00.yyy.0/cc', 'CidrBlockSet': [KVPDict({'CidrBlock': 'xx.00.yyy.0/cc'})]})

       

       

      Now if I modify the logic with more nested loops to parse KVPDict, I am getting error mentioned below. My understanding was KVPDict is not simple string.

       

      target_groups := discovery.AWS.EC2.DescribeVpcPeeringConnections(da);

      for item in target_groups do

           for key in item do

                if key <> "__index" then

                     val_tal := item[key];

                     log.info("%key%:%val_tal%");

                     for item1 in val_tal do

                          for key1 in item1 do

                               val_tal_1 := item1[key1];

                               log.info("#####>>>>>%key1%:%val_tal_1%");

                          end for;

                     end for;

                 end if;

           end for;

      end for;

       

      (0): 266ms: Status:KVPDict({'Message': 'Active', 'Code': 'active'})

      (0): 275ms: RuleError on rule tpl_VPCPeering_body_1 due to: Error while executing a rule -- TypeError: string indices must be integers, not str

       

      I am not sure how to move ahead with this. Please help.

        • 1. Re: AWS - AWS.EC2.DescribeVpcPeeringConnections output parsing
          Andrew Waters

          Treat a KVPDict as a table - that is how it will work within TPL.

           

          What you are doing is not going to work. Trying to do a generic thing like this is going to fail because Tags is a list not a KVPDict, VpcConnectionPeeringId is a string not a KVPDict.

           

          Pick out the fields that you need and process them specifically.

          1 of 1 people found this helpful
          • 2. Re: AWS - AWS.EC2.DescribeVpcPeeringConnections output parsing
            Amrendra Kumar Singh

            I must populate the Tags value for each resource under a resource group.

             

             

            As per my understanding we can achieve using two methods.

            1. Write a pattern to trigger on each resource type example AdminCollection node, then search for DiscoveredCloudAPIResult for related resource and then populate the “Tags” attribute in AdminCollection with Tags value present in the Resource (DDD).

             

            This is working fine for me.

             

                triggers

                    // Trigger on ListResourceGroups Cloud API result list

                    on result_list := AdminCollection created, confirmed where cloud exists;

                end triggers;

                       

                       

                          body

                       

                                        result := search(in result_list traverse InferredElement:Inference:Primary:DiscoveredCloudAPIResultList traverse List:List:Member:DiscoveredCloudAPIResult where $tags defined and name = %result_list.name% show $tags);

                                        // where $tags defined and name = "AZURE_API_PROD" show $tags ----  $tags defined show $tags

                                     

                                        size_val := size(result);

                                        log.info("Size is: %size_val%");

                                        log.info("%result%");

                                     

                                        if result then

                                                      result_list.Tags := result;

                                                      //log.info("%result_list.Tags%");

                                                      model.addDisplayAttribute(result_list, "Tags");

                                        end if;

             

            2. Out of curiosity, I am looking into second method where, I am triggering pattern on DiscoveredCloudAPIResultList where discovered method is "Azure.ResourceGroups.ListAll".

             

            I used getResults to get the individual DiscoveredCloudAPIResult, extracted the Resource name, tag and srubscriptionID.

             

            Using name and subscription Id I searched for the AdminCollection and the tried to use the extracted TAG value to populate in AdminCollection, but it is failing with error mentioned below:

            RuleError on rule tpl_InferTags_body_1 due to: Error while executing a rule -- TypeError: 'NodeSetProxy' object does not support item assignment

            I have attached there pattern herewith.

             

            To me the output of search query in point one looks the same as the type as of value extracted for tags attribute in point 2 i.e. KVPDict, so I am not able to understand why second approach is giving me error.

            • 3. Re: AWS - AWS.EC2.DescribeVpcPeeringConnections output parsing
              Andrew Waters

              Your problem is

              vm := search(in result_list

                              traverse Primary:Inference:InferredElement:CloudService

                              traverse Collection:Collection:Member:AdminCollection

                              where type = "Microsoft Azure Resource Group"

                                  and name = "%api_results.name%"

                                  and subscription_id = "%sub_id_regex_val%");

               

              vm.Tags := api_results.$tags;

              because vm is a set of nodes. Even if there is one result it is a set of nodes. If you expect one node then you can do something like

              if vm then

                vm[0].Tags := api_results.$tags;

              end if;

              which checks that the search actually found something and gets the first node out of the set of nodes.

              1 of 1 people found this helpful
              • 4. Re: AWS - AWS.EC2.DescribeVpcPeeringConnections output parsing
                Amrendra Kumar Singh

                As discussed above, using below mentioned method I can assign the Tags information from ResourceGroup to AdminCollection node or VM. But now I have two issue.

                1. I cannot extract the report with value for Tags.

                2. I am not able to sync Tag value in CMDB.

                ==================================================================================================

                result := search(in result_list traverse InferredElement:Inference:Primary:DiscoveredCloudAPIResultList traverse List:List:Member:DiscoveredCloudAPIResult where $tags defined and name = %result_list.name% show $tags)                        

                                          if result then

                                                          result_list.Tags := result;

                                                          model.addDisplayAttribute(result_list, "Tags");

                                            end if;

                ===================================================================================================

                 

                As tags are of data type which I am treating like table.

                 

                [[KVPDict({'Environment': 'DevTestLabs', 'Division': 'UUU', 'consumer': 'abc.ret@abc.com', 'Costcenter': '00000000003', 'Application': 'abc Cloud'})]]

                 

                I have assigned Tags from ResourceGroup to Tags attribute in Azure Virtual machine node, now when I try to extract report, it gives me report

                  

                VM TypeNameInstance CountProduct VersionContained HostTags (Tags)
                Microsoft Azure VMMicrosoft Azure VM azusirwd01 in East US1NoneNone[[KVPDict({'Environment': 'DevTestLabs', 'Division': 'UUU', 'consumer': 'abc.ret@abc.com', 'Costcenter': '00000000003', 'Application': 'abc Cloud'})]]

                 

                How to get reports which would give me individual tags?

                 

                I am facing similar issue with populating the attributes in CMDB for corresponding individual tags.

                 

                If there is no way out of this then, it seems I have to create separate attribute for each tag by using steps mentioned below.

                 

                            if tag_key  then

                                new_tag_attr_name := "tag_" + tag_key;

                                node[attr_name] := tag.Value;

                            end if;