Manage Grepr Pipelines as Code with the Terraform Provider

Starting from a pipeline built in the Grepr UI, Michael shows the full round-trip: Export the pipeline's job graph JSON with the Grepr CLI and jq Import an existing pipeline into Terraform state with an import block (provider version 1.1) Run terraform plan and apply to bring the pipeline under Terraform management Make a change in Terraform (updating a reducer's similarity threshold) and watch it apply to the live pipeline in the UI Reconcile a change made in the UI back into your Terraform configuration so state and server stay aligned By the end you've seen a complete cycle: UI to Terraform, Terraform to UI, and UI back to Terraform. Transcript: Hi, my name's Michael. I'll be walking through the Grepr Terraform provider today. Here on our UI, we have a demo pipeline. It was created in the UI, and we'll use this as an example for how you can make updates and bring it into Terraform, and vice versa, make edits in Terraform and see them in the UI. You can see there's some configuration. Right now we have a source set up with no logs flowing through, and there's no sync, so nothing is being written anywhere, since this is just for our demo. If we go to my terminal, we can look at the JSON that backs this job using the Grepr CLI. If you run the job get command, it outputs the JSON of a job. However, what actually backs the Terraform provider is just the job graph JSON, so we do some jq magic to get the job graph into its own file. The most up-to-date version of the Terraform provider is 1.1, and I suggest using that one. You can see an example of what our Grepr pipeline resource looks like here, and we'll use this import block to import the pipeline from our JSON into the Terraform state. This is an example of what the pipeline JSON looks like. There's a bunch of configuration here for the whole pipeline. We received this via the Grepr CLI and saved it into this pipeline.json file, and it gets referenced in the Grepr pipeline resource. Now if we run terraform plan, we should see the plan for the demo pipeline getting the configuration into our Terraform state. There won't be any actual changes applied to the pipeline in this step. This is essentially just importing the pipeline into the Terraform state so it can be tracked. The plan is going to show a large diff, essentially the entirety of the pipeline configuration. Since this is the first import, it shows all of the configuration along with some metadata at the bottom. Now if I run terraform apply, it will import this pipeline into the Terraform state, and everything will be in sync. The apply is complete and it has been imported. So now we can do another terraform plan and see that we're in sync between our Terraform state and what is defined on the Grepr server. Now let's test making a change via the pipeline.json file that backs this Terraform resource, then apply the change, and we should see the resulting pipeline change in the UI as well. The configuration we're going to change is a reducer configuration, updating the similarity threshold from 70% to 85% as our example. If we apply this change, it performs the actual update. It can take a few seconds up to a minute. If we go back to the UI, we'll see the pipeline in its transitioning state while it performs the update. Back in the terminal, the apply completed, and back in the UI, we can check the reducer step and see it has the new value, 85%. Now if we edit the pipeline in the UI again, this will put the pipeline out of sync with what we have defined in the Terraform state. Terraform will continue to say we have a similarity threshold of 85, whereas after this update the actual pipeline will be running at 60%. That's okay, there won't be a problem, but we'll have to update what we have defined in the Terraform resource to match what's on the server, so that the next time someone runs terraform apply there isn't an issue with the desired state. If we run terraform plan right now, it will try to re-correct the similarity threshold from 60 back to 85. But if we bring the new pipeline JSON from the Grepr server into the pipeline.json file, it will correctly show 60%, which is what we want. We'll use jq to get the job graph JSON back into our pipeline.json file, and we'll see the new similarity threshold is 60. Ideally at this point, if we ran terraform plan and apply, it would say no changes. However, we have a parsing bug in our Terraform provider that maps that 60 to a double, 60.0, so it shows a difference. If we run terraform apply just to get the Terraform state to say no changes, it corrects it in Terraform state, so there's no actual update to the pipeline. The next time you run terraform plan after this apply, it will be correct from Terraform's point of view. And now we've done a full cycle, going from UI to Terraform, Terraform to UI, and UI back to Terraform. Thanks for sticking around. Bye.