👋 Hi! I’m Bibin Wilson. In each edition, I share practical tips, guides, and the latest trends in DevOps and MLOps to make your day-to-day DevOps tasks more efficient. If someone forwarded this email to you, you can subscribe here to never miss out!

✉️ In Today’s Edition

In today’s edition, we’ll look at Steampipe, an open-source tool that lets you query your cloud infrastructure, Kubernetes resources, Terraform configurations, and more using SQL.

We’ll cover:

  • What is Steampipe and how it works

  • Practical use cases for DevOps and cloud engineers

  • Hands-on examples using SQL queries

and more..

Note: For a better reading experience, read the online version.

📦 Latest DevOpsCube Guides

  1. Best AI/ML Certifications for DevOps: AI/ML is one of the most sought-after skills in the IT industry today. Here is where AI/ML certifications come into play. They provide a structured learning path for AI/ML technologies and implementations.

  2. EKS Cluster Autoscaler Setup: In this hands-on guide, you will learn to implement node autoscaling in an EKS cluster using Cluster Autoscaler.

  3. Image Signing and Attestations in Kubernetes: With the help of the hands-on exercises in this tutorial, we will help you understand how container image signing and attestation works in Kubernetes.

Use Case

How many times have you opened the AWS console and other tools just to answer a simple question like:

  • Which resources don't have the required labels?

  • Are there any storage buckets that can be accessed publicly?

  • Which Kubernetes namespaces don't have NetworkPolicies?

  • Which Terraform resources are missing encryption settings?

The answers are there, but finding them often means jumping between services, checking multiple accounts, or writing scripts.

What if you could answer these questions using a simple SQL query?

That is exactly what Steampipe helps you do.

What is Steampipe?

Steampipe is an open-source tool that lets you query your cloud infrastructure using SQL as shown in the image below.

img src: steampipe.io

Instead of navigating through multiple pages in a cloud console, you can ask questions about your infrastructure using simple SQL queries.

For example, you can quickly find publicly accessible S3 buckets across your AWS accounts.

Who is Steampipe for?

Steampipe is mainly useful for people who need to answer questions about cloud infrastructure rather than constantly monitor it.

  • Cloud security engineers can use it for quick security audits, compliance checks, and gathering evidence without manually going through the AWS console.

  • DevOps and platform engineers can use it to get a quick inventory of infrastructure across regions and accounts, or build simple checks into their workflows.

  • FinOps teams can use it to find resources that are costing money without providing much value, and combine cost data with information about the resources themselves.

It is also useful for consultants, auditors, or anyone who has just inherited a cloud account and needs to understand what is actually running in it.

How Does Steampipe Work?

The following diagram shows how Steampipe lets you query cloud and Kubernetes resources using SQL.

Steampipe runs a lightweight PostgreSQL database locally and connects to your cloud using your existing credentials.

When you run a SQL query,

  1. Steampipe receives the SQL query.

  2. The relevant plugin translates it into cloud API calls.

  3. The cloud provider returns the requested resource data.

  4. Steampipe maps the data into rows and columns.

  5. PostgreSQL returns the result to your terminal.

Under the hood, Steampipe plugins use PostgreSQL Foreign Data Wrappers (FDWs). This allows external resources such as AWS, Azure, GCP, and Kubernetes to appear as regular database tables.

For example, the AWS plugin translates your SQL query into calls to the relevant AWS APIs.

Setup Steampipe & Plugins

To get started, you need to install Steampipe on your local machine.

## MAC OS

brew install turbot/tap/steampipe

## Linux

sudo /bin/sh -c "$(curl -fsSL https://steampipe.io/install/steampipe.sh)"

Verify the installation.

steampipe -v

Steampipe doesn't know anything about AWS or GCP out of the box. You install a plugin for whichever provider you want to query.

You can find all the supported plugins here.

For example, in my case I am using AWS and Kubernetes. You can install the plugins using the following commands.

$ steampipe plugin install aws

$ steampipe plugin install kubernetes

Note: The AWS plugin uses the AWS SDK, not the AWS CLI. But it uses the same standard AWS credential chain, so it can pick up credentials from environment variables, AWS profiles, SSO, and IAM roles depending on where it is running.

Now, you can run a one-off query straight from your terminal.

steampipe query "select name, region from aws_s3_bucket"

Use the following command to enter the query mode. It drops you a prompt where you can type queries and press Enter to run them:

steampipe query

Note: Press Ctrl+D or type .exit to leave the query mode

Practical Steampipe Examples

Now, let’s look at some practical use cases for Steampipe.

Important Note: You can find the available tables, columns, and query examples in the documentation for each Steampipe plugin. For example, refer to the AWS plugin tables for AWS resources and the Kubernetes plugin tables for Kubernetes resources.

1. Cloud Security Checks

In cloud security, questions like "is there anything exposed that shouldn't be?"
are very common. Steampipe is built to answer these kinds of questions.

Finding all publicly accessible buckets across your account generally means checking each one individually or using services such as AWS Config.

With Steampipe, you can run the following query and get the answer in seconds.

> select name, region
from aws_s3_bucket
where bucket_policy_is_public = true;


+-------------------------------------------+-----------+
| name                                      | region    |
+-------------------------------------------+-----------+
| static-web-arun-637423664276-us-west-2-an | us-west-2 |
+-------------------------------------------+-----------+
1 row

Similarly to find users without multi-factor authentication you can run the following query.

select
  name,
  mfa_enabled
from
  aws_iam_user
where
  mfa_enabled = false;

These kinds of checks are run regularly run by the security teams.

2. FinOps and Cost Visibility

Steampipe’s AWS plugin exposes Cost Explorer data as tables, including cost by account, service, region, tag, forecasts, and usage.

For example, you can find which services are driving the most cost using the following query.

select
  service,
  period_start,
  unblended_cost_amount
from
  aws_cost_by_service_monthly
where
  period_start > now() - interval '3 months'
order by
  unblended_cost_amount desc
limit 10;

3. Kubernetes Configuration Auditing

In a Kubernetes cluster, you may want to know which containers are running without CPU or memory limits.

For example, to find containers without memory limits:

select
  name as pod_name,
  namespace,
  c ->> 'name' as container_name
from
  kubernetes_pod,
  jsonb_array_elements(containers) as c
where
  c -> 'resources' -> 'limits' ->> 'memory' is null;

That’s a Wrap!

The main advantage of Steampipe is that you can answer questions about your infrastructure without writing a script every time.

Need to find public S3 buckets? Write a query. Need to check which instances are missing tags? Query it. Need to look across multiple AWS accounts or combine data from different services? It’s the same SQL interface.

Does it replace AWS Config, CSPM platforms, or your monitoring stack?

No. It gives DevOps, platform, security, and FinOps teams a simple way to explore and query infrastructure when they need answers.

Reply

Avatar

or to participate