I don't use US providers if I don't have to. I will do a post on why some time in the future. Because AWS S3 has been such a success many tools have created support for the S3 object storage. So, both OpenStack Swift and Ceph have built compatability for the S3 API. And Ceph has the benefit of being compatible with both S3 API and OpenStack Swift API.

Ceph is the object storage solutuion at Cleura. They have quite good documentation on how to get get started. Like how to create ec2 credentials with OpenStack, so I suggest you follow their guide.

To use S3 backend with OpenTofu you first need to create a bucket. Then you can configure the backend and initialize with OpenTofu. And from there on you are rocking a remote backend set up with OpenTofu.

In OpenTofu the configuration needed is not not complicated. The snippet below shows the configuration needed for OpenTofu:

terraform {
  required_version = ">= 1.8.0"
  required_providers {
    openstack = {
      source  = "terraform-provider-openstack/openstack"
      version = "~> 2.1.0"
    }

  }

  backend "s3" {
    bucket = "a-cool-bucket-name"
    key    = "a_cool_state_name.tfstate"
    endpoints = {
      s3 = "https://s3-kna1.citycloud.com"
    }
    skip_s3_checksum            = "true"
    region                      = "us-east-1"
    use_path_style              = "true"
    skip_credentials_validation = "true"
    skip_requesting_account_id  = "true"
    skip_metadata_api_check     = "true"
    access_key                  = var.s3_access_key
    secret_key                  = var.s3_secret_key
  }
}

The acces_key and even more so the secret_key should be provided in a secure way. Either by your OpenTofu runner of choice or using Terragrunt and SOPS or something like that.

At Cleura the Karlskrona datacenter, Kna1, has an Object Storage with S3 compatability with features such as Object-Lock, Object-Versioning and Server Side Encryption (SSE).

Since I am using OpenTofu, I want to use state encryption later. That means I will encrypte the state file on the client side and thus protecting the state file before it reaches the object storage. The benefit of that is that I will not need to provide the encryption key to Cleura.

To create a bucket you might use the AWS S3 cli tool. The following snippet shows how to create a bucket with object lock enabled which also enables object versioning.

aws --profile PROFILENAME \
  s3api create-bucket \
  --bucket BUCKETNAME \
  --object-lock-enabled-for-bucket

When you have created the bucket and the backend configuration is complete you can perform tofu init.

If you already have a state file, make sure it is located inside your project root folder, called terraform.tfstate and perform tofu init -migrate-state

In a future post I will dig a bit deeper into state encryption and backups of the state file and how to use SOPS with OpenTofu.

#terraform #opentofu #openstack #s3 #objectstorage #remotebackend #cleura

Joakim Durehed

Regrettably, I have to work with Windows Server every day. I won't go into why I think this is regrettable in this post. But, one, of many, thing that sucks when running Windows Server is that so much information on the amazing world wide web is geared towards running applications on some sort of Linux. In the “put something on a server” world, some kind of Linux is the norm.

Anyway, I wanted to create a local Administrator account without having to push a password through CloudBase-Init in the user_data configuration. As an example one can do something like this:

users:
  - name: cool_boy_82
    groups: Administrators
    passwd: very-cool-and-strong-password
    ssh_authorized_keys:
      - ssh-rsa AAAB...byV

However, even if the above do work you get a little bit of a security issue with this approach. Say that you forget to reset the password you give user cool_boy_82 and you have some kind of breach. It can be possible to either read the logs from CloudBase-Init in C:\Program files\Cloudbase or to fetch the logs from the metadata service. Because if you put the password in clear text, it will be available in the logs in clear text. You might use this as a default account that you always add when you create new instances in OpenStack. Well, then you have a security issue.

Let CloudBase-Init generate user password

One solution for this is to let Cloudbase-Init to do the password generation for you. Then you retrieve it later through the magic of encryption. However, I ran into some issues. Whenever I tried to retrieve the password from the OpenStack CLI I got the following error:

b'unable to load Private Key [...] PEM routines [...]

So, something was wrong. And as so many times before, the error were somewhere behind the keyboard.

When you create an instance in OpenStack you tell it to use a ssh-key public key of your choice. The ssh-key public key can be used to encrypt the password that Cloud-Base Init generates so that ONLY the one who has access to the private key can decrypt it.

It is quite easy, but I ran in to a gotcha. Turns out that it is very important to create the ssh-key in a specific format and type of key. Also, if you password protect your private key, you can't retrieve the password from the OpenStack Horizon UI.

We need to use a ssh-key with type rsa and make sure that it is in the pem format. I use ed25519 as my type of choice. Especially after I started using a TKey from Tillitis wich only generates ed25519 keys.

So to generate a ssh-key of the correct type and format the following should suffice:

ssh-keygen -t rsa -m PEM -f "cool_boy_82"

So, then to retrieve the password generated for an instance use the OpenStack CLI with the following command:

nova get-password {instanceID} {PathTo: private key file}

So, all is well. If you simply do things the right way things work better. One thing that I do feel is not so “well” is the documentation from OpenStack and some of the OpenStack providers. I did search a hell of a lot to find information about this issue and I could not find conclusive, to me at least, information about what OpenStack required in terms of ssh-keys and formats. One good reasource is from the Swedish OpenStack provider Safespring. They have a very good blog post about ssh-keys and OpenStack.

Anyway, I put this out on the wide wide wide web to let any body else have the chance to find this quicker than I did.

#openstack #sshkey #cloudbaseinit #cloudinit #tkey #tillitis

Joakim Durehed

I really like that there is a free and Open Source fork of Terraform now. For me, the future is Open Source, and specifically Free and Open Source.

I don't believe this because I like to not have to pay for software. I believe this because it is no other way to move forward with digital transformation at a high pace effectively. Very much so in the public sector, but also for organizations in general.

OpenStack is my platform of choice. First off, it is Open Source, and second off, a lot of public cloud providers deploy OpenStack. So there are a lot of places to put your infrastructure.

And this is where a cool thing about infrastructure as code comes through. It is so easy to deploy resources to different providers and platforms that it was almost lost on me. I had to try it out and then I started to get it.

But, anyhow, lets do something with OpenTofu and OpenStack.

The basic stuff first

We are doing this without extra tools like Terragrunt and Gitlab. I usually create a variable to hold all parameters to connect to OpenStack. I keep the sensitive part, the application credential secret in a separate file called connect.auto.tfvars. This way, that file, containing the secret will be read by OpenTofu at plan, apply, destroy and so on.

But, when working in a team, this would preferably be read from a secret manager of some sort or in your build server so that you don't even need to store the secret locally.

Below is the file I call main.tf because it is the starting point in my mind with the provider connection parameters and the OpenTofu configuration.

terraform {
  required_version = ">= 0.14.0"
  required_providers {
    openstack = {
      source  = "terraform-provider-openstack/openstack"
      version = "~> 1.53.0"
    }
  }
}

provider "openstack" {
  application_credential_id     = var.connection.application_credential_id
  application_credential_secret = var.application_credential_secret
  auth_url                      = var.connection.auth_url
  region                        = var.connection.region
  endpoint_type                 = var.connection.endpoint_type
}

To use the main.tf file we also need a variables.tf file. You could call this whatever you want, but I will probably add more variables to this file, so instead of calling it connect.tf to categorize it with the connect.auto.tfvarsfile, I just call it varibles.tf.

I use Binero as the public cloud provider. Binero is based in Sweden, as am I.

I use an application credential instead of a user account because I like to differentiate using user accounts from API oriented accounts. Application credentials were designed for this purpose.

variable "connection" {
  type = object({
    application_credential_id = string
    auth_url                  = string
    endpoint_type             = string
    region                    = string
  })

  default = {
    application_credential_id = "application_id"
    auth_url                  = "https://auth.binero.cloud:5000/v3"
    region                    = "europe-se-1"
    endpoint_type             = "public"
  }
}

variable "application_credential_secret" {
  type      = string
  sensitive = true
}

Create a network

So with a main.tfvars and variables.tf accompanied with the connect.auto.tfvars file we have all we need to make some resources with OpenTofu.

In this case I will create a network and a subnet to that network.

I add a variable for the network cidr in the variables.tf. This could be more comprehensive, but works for our example.

variable "main_network_cidr" {
  type    = string
  default = "10.10.10.0/24"
}

And we use the above variable in the resource declaration in network.tf:

resource "openstack_networking_network_v2" "main" {
  name           = "main"
  description    = "Main network for general purpose."
  admin_state_up = "true"
}

resource "openstack_networking_subnet_v2" "main" {
  name        = "main"
  description = "Main network subnet for general purpose."
  network_id  = openstack_networking_network_v2.main.id
  cidr        = var.main_network_cidr
  ip_version  = 4
}

I would normally create input variables for all the above arguments in the resource blocks. But again, this is just an example. The idea of using a variable for each argument is that you then can re-use the resource block in other projects and create a module of it. And it makes it possible to add variables when performing tofu apply.

The above should get you running with creating a network and subnet in OpenStack. Remember that the connection arguments might differ between OpenStack providers.

Also, this is just an example, I will create more comprehensive projects in the future where I will gear towards something that is production ready. I will also share my code in a public repository (as soon as my public repository is ready for that)

I hope you found this useful. As stated, more is to come.

#terraform #opentofu #openstack #network

Joakim Durehed

Today is the first time I use OpenTofu the Open Source Terraform contender.

HashiCorp made the terrible decision to go for a less open license called BSL. A license aimed at stifle any competition on creating services on top of the products under this license. In this case, it will make it impossible to create Terraform Cloud competitor for instance.

It is of course not illegal for Hashicorp to do this. They are completely free to change their license. They should have expected the backlash. And I think they did since they have not said a word about it really since the license change as far as I can tell.

I might do a write up on why a license change like this is a shitty move, but for now I'll point you to this good write up on that.

Back to OpenTofu. The first release, OpenTofu 1.6, is compatible with Terraform 1.6. They have made it very, very simple to install and to migrate from Terraform to OpenTofu. I'm not migrating anything today but simply creating some stuff with OpenTofu for the fun of it. And to just start using it.

You still use the .tf file ending and the provider configuration block is still named Terraform as you can see below.

terraform {
  required_providers {
    openstack = {
      version = ">= 1.54.1"
      source  = "terraform-provider-openstack/openstack"
    }
  }
}

What I lack today is a a way to run tofu apply with a remote state. There are some tools that do this, but most do it a way that really don't fit how I work right now. Of course this can be done with Terraform using Terraform Cloud. Which I do like and use to some extent. But, it of course comes with a price in different forms. For one, I can only us Terraform, not OpenTofu. It runs on AWS if I'm not mistaken and that is not really my taste (GDPR/FISA). And it also ties me to Hashicorp and their willingness to increase prices.

Some times it is a good fit to use a SaaS and just pay as you go. But some times, it is not. For me the problem is that I need to store critical credentials in Terraform Cloud. Credentials that allows for Terraform to create, modify and delete resources. I'm not saying that Terraform Cloud is not secure. It is probably very, very secure. But every something-as-a-service you use adds to the list of suppliers you need to trust. And even though I do trust Hashicorp to be very security first minded and extremely competent. I mean, really, competent. They are extremely talented. Period. No one is immune to attacks or mistakes.

So, as silly as it might sound to some, but when it comes to this level of trust, I rather be the one who do the mistakes than others. That is because when shit hits the fan, which it will at some point, I will have most if not all the pieces to the puzzle of what happened. And that to me is extremely important. I don't need anyone to blame.

Anyways, I will create OpenTofu related content here that is more practical and not like this post, a bunch of mumbling.

I also want to make it clear that even though I don't like the choice Hashicorp has made in regards to licensing. I still really like what they do in general and have so much to thank them for. They have made my job so much better by creating Packer, Vagrant, Terraform and so on.

#opentofu #terraform #openstack

Joakim Durehed