Affichage des articles dont le libellé est AWS. Afficher tous les articles
Affichage des articles dont le libellé est AWS. Afficher tous les articles

lundi 15 janvier 2024

terraform, then ansible

Terraform creates the infra, 
then we want to use ansible to actually configure it...

Different solutions exist to run Terraform, and them ansible:

  • Using Terraform Output as Ansible Inventory :
    https://github.com/adammck/terraform-inventory
     $ terraform-inventory -inventory terraform.tfstate [all] 10.10.1.2 10.10.1.3


  • `local-exec` / `remote-exec` : terraform runs ansible locally 

     ``` provisioner "local-exec" {command = "ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -u {var.user} -i '${self.ipv4_address},' --private-key ${var.ssh_private_key} playbook.yml"} ```
    key component here is the ${self.ipv4_address} variable. After provisioning the machine, Terraform knows its IP address. And we need to pass an IP address for Ansible.

     (cf https://www.cprime.com/resources/blog/terraform-and-ansible-tutorial-integrating-terraform-managed-instances-with-ansible-control-nodes )

     

  • using dynamic inventory and cloud providers specific ansible modules 

    •  AWS : https://docs.ansible.com/ansible/latest/collections/amazon/aws/aws_ec2_inventory.html 
    •  GCP : https://docs.ansible.com/ansible/latest/collections/google/cloud/gcp_compute_inventory.html#ansible-collections-google-cloud-gcp-compute-inventory and so on...



AWS example of the dynamic inventory:


ansible.cfg :
enable_plugins=aws_ec2

aws_ec2.yml (example):
plugin: aws_ec2
regions:
  - "us-east-1"
keyed_groups:
  - key: tags.Name
  - key: tags
    prefix: tag
  - prefix: instance_type
    key: instance_type
  - key: placement.region
    prefix: aws_region
filters:
  instance-state-name : running
  # All instances with their `Environment` tag set to `dev`
  tag:Environment: dev
  # All dev and QA hosts
  tag:Environment:
    - dev
    - qa
compose:
  ansible_host: public_ip_address



Good way to test : 
ansible-inventory -i aws_ec2.yml --graph
ansible all –list-hosts



To run a playbook have/generate 4 files : myplaybook.yml, key.pem, aws_ec2.yml (seen before), and ansible.cfg (seen after)


ansible.cfg
[defaults]
inventory=./aws_ec2.yml
host_key_chekcing=false
remote_user=ec2-user
private_key_file=key.pem

[privilege_escalation]
become=true
become_method=sudo
become_user=root


Run 

ansible <group-name> -i aws_ec2.yaml -m ping --private-key=<private-key-name> 

ansible-playbook myplaybook.yml
 (with hosts: _Ansible_TargetNode or whatever you have validated in the output of the inventory)






# sources : 
https://www.cloudthat.com/resources/blog/step-by-step-guide-to-integrate-ansible-dynamic-inventory-plugin-for-aws-ec2-instances#why-ansible-dynamic-inventory-

https://medium.com/geekculture/a-complete-overview-of-ansible-dynamic-inventory-a9ded104df4c



mercredi 17 février 2021

Run Ansible playbook on AWS target with SSM System manager

AWS configuration

SSM State manager : Association Parameters

 documentParameters with an archive (zip) containing multiple yml files

{ "InstallDependencies":"False", "SourceType":"S3", "SourceInfo":"{\"path\":\"https://name_of_bucket_hosting_sources.s3-eu-west-42.amazonaws.com/prefix_key/archive.zip\"}", "PlaybookFile":"main.yml" }

 documentParameters with only one yml files

{ "InstallDependencies":"False", "SourceType":"S3", "SourceInfo":"{\"path\":\"https://name_of_bucket_hosting_sources.s3-eu-west-42.amazonaws.com/prefix_key/playbook.yml\"}", "PlaybookFile":"playbook.yml" }

sourceInfo

{ "name": "AWS-ApplyAnsiblePlaybooks" }

* Association Target

Depending on where you want to run the playbook, select what's appropriate




Ansible playbook 

example : daily export from an EC2 instance directory to an s3 bucket


- hosts: localhost
  become: yes
  become_method: sudo
  tasks:
  - name: Find zips in /path/to/data/*.zip older than 7d
    find:
      paths: /path/to/data/
      patterns: '*.zip'
      age: 7d
    register: files_to_delete
  - name: Remove zips in /path/to/data/ older than 7d
    file:
      path: "{{ item.path }}"
      state: absent
    with_items: "{{ files_to_delete.files }}"

  - name: Upload content of /path/to/data/ directory, ommiting structure-*.zip files
    community.aws.s3_sync:
      bucket: target-s3-share-name
      key_prefix: s3-prefix-dir-name/subdirectory/
      file_root: /path/to/data/
      include: "*.zip"
      exclude: "structure-*.zip"
      delete: no # if set to yes, removes remote files that exist in bucket but are not present in the file root.
  - name: Upload content of /path/to/data/ directory
    community.aws.s3_sync:
      bucket: target-s3-share-name
      key_prefix: s3-prefix-dir-name/subdirectory/
      file_root: /path/to/data/
      include: "*"


References :

mercredi 25 novembre 2020

OpsGenie : AWS SNS message to Jira Description Wiki markup (+ links to S3 logs and SSM output)

If you're using the AWS SNS opsgenie integration and want to publish to JIRA, you can for example use the following code to present the data in a slightly better way : 


I this use-case I'm using the SNS channel to publish outputs from a system manager (AWS SSM) command that also publishes it's outputs to an S3, so we're using this extraction to provide the direct links to the s3 logs and the SSM run command history.

And in the end, we copy the message we received from the SNS channel "raw"..


In Opsgenie, in the specific Amazon SNS integration (Incoming Amazon SNS), in the Alert Fields, you can for example modify the "Description" so that it transforms the Message received like this :


h3. Details
|| AWS region | {{ TopicArn.extract(/arn:aws:sns:([^:]*):.*/) }}  |
|| Status | {{ Message.extract(/.*"status":"([^"]*)".*/) }} |
|| Instance ID |   {{ Message.extract(/.*"instanceId":"([^"]*)".*/) }}  [(aws link)|https://{{ TopicArn.extract(/arn:aws:sns:([^:]*):.*/) }}.console.aws.amazon.com/ec2/v2/home?region={{ TopicArn.extract(/arn:aws:sns:([^:]*):.*/) }}#InstanceDetails:instanceId={{ Message.extract(/.*"instanceId":"([^"]*)".*/) }}]|
|| Command ID | {{ Message.extract(/.*"commandId":"([^"]*)".*/) }} [(aws cmd)|https://console.aws.amazon.com/systems-manager/run-command/{{ Message.extract(/.*"commandId":"([^"]*)".*/) }}]  [(s3 logs)|https://console.aws.amazon.com/s3/buckets/ssm-output/ssm-log/{{ Message.extract(/.*"commandId":"([^"]*)".*/) }}/{{ Message.extract(/.*"instanceId":"([^"]*)".*/) }}/?region={{ TopicArn.extract(/arn:aws:sns:([^:]*):.*/) }}&showversions=false ]  
|
|| documentName | {{ Message.extract(/.*"documentName":"([^"]*)".*/) }} |
|| requestedDateTime | {{ Message.extract(/.*"requestedDateTime":"([^"]*)".*/) }} |
|| eventTime | {{ Message.extract(/.*"eventTime":"([^"]*)".*/) }} |
h3. Opsgenie info
|| EventType | {{eventType}} |
|| Timestamp (opsgenie) | {{Timestamp}}|
|| Tags | {{tags}} |
|| TopicArn | {{TopicArn}} |
|| Actions | {{actions}} |
h3. Original Message (raw): 
{code}
{{Message}}
{code}



Nb: this might only be available in certain OpsGenie subscriptions unfortunately :-(