Custom service to service authentication using IAM/KMS

Similar to abusing IAM and STS, we can also abuse IAM and KMS to let Amazon do our service-to-service authentication for us. Unlike STS, though, KMS is almost perfect for this use case.

Let’s recap a bit from the STS post, though. What I’m aiming for is service to service authentication with the following specs:

  1. Has no chicken and egg trust problem. Re-use AWS to provide the chicken, we’ll use it to lay the eggs.
  2. Can be used only from one service to another service. The service receiving the token shouldn’t be able to reuse the token to impersonate the sender.

Custom service to service authentication using IAM/STS

Lately I’ve wanted to be able to use IAM directly for authentication. Specifically, I wanted a way for a service to be able to verify that a request from another service was from a particular IAM role and that the request’s auth is still valid. I want this because I want to avoid the chicken and egg problem of bootstrapping auth systems in the cloud. You need some secret on your instances that allow it to talk to an auth system to do authentication between services. Amazon already provides this in the form of IAM roles. Unfortunately, only AWS services have access to verify these credentials. However, it’s possible to abuse IAM and STS to achieve this goal.

Using development branch SaltStack python modules in the stable release

SaltStack has a lot of development work occurring and sometimes you want early access to features. Thankfully, for a number of Salt’s features it’s possible to very easily bring them into the stable release without needing to maintain a fork or a non-standard version.

Salt is built to be very modular, so that you can write/run your own custom modules. This same mechanism works for overriding the core modules as well. I’ll explain how this works in a masterless setup (since that’s what I use), but the process is pretty similar for master/minion setups as well.

KMS support added to SaltStack Beryllium (development) branch

In the development branch of SaltStack (to be the Beryllium release) I’ve added the boto_kms state and execution modules. This allows you to manage KMS master keys, their policies, key rotation, and other attributes via states. It also allows you to make KMS calls from other state and execution modules.

Here’s an example of managing a key and its attributes via a state:

Investigating local queuing: Redis, NSQ and LMDB

Systems designed for cloud services assume instances can die at any time, so they’re written to defend against this. It’s important to also remember that networks in cloud services are also incredibly unreliable, and often much less reliable than the instances themselves. When considering a design, it’s important to remember that a node can be partitioned from other services and possibly for long periods of time.

One easy consideration here is logs (including stats and analytics events). We want to ensure delivery of logs, but we also don’t want delivery to affect service operation.

DynamoDB support in SaltStack 2015.2

In the 2015.2 SaltStack release we’ve added the boto_dynamodb execution module and boto_dynamodb state module. This allows you to create DynamoDB tables via states:

Ensure DynamoDB table exists:
    - table_name: {{ grains.cluster_name }}
    - read_capacity_units: 10
    - write_capacity_units: 10
    - hash_key: id
    - hash_key_data_type: S
    - global_indexes:
      - data_type_date_index:
        - name: "data_type_date_index"
        - read_capacity_units: 10
        - write_capacity_units: 10
        - hash_key: data_type
        - hash_key_data_type: S
        - range_key: modified_date
        - range_key_data_type: S
      - data_type_revision_index:
        - name: "data_type_revision_index"
        - read_capacity_units: 10
        - write_capacity_units: 10
        - hash_key: data_type
        - hash_key_data_type: S
        - range_key: revision
        - range_key_data_type: N

Note that at this point the module will only create and delete tables and indexes. It doesn’t currently support dynamically adding and removing indexes or changing read or write capacity units. These are features we’d love to see, and will likely add in the future. If you’d like to beat us to this, please send in pull requests!

Splunk saved search state and execution module support in SaltStack

We (Lyft) believe strongly in the concept of infrastructure as code. If it isn’t automated, it isn’t finished. This belief also applies to our monitoring and alerting. We’re using Splunk saved searches for portions of our alerting and want to ensure that our developers can quickly and easily define alarms in a standard way to be able to share alarms between services.

We’ve added the splunk_search execution module and splunk_search state module to the 2015.2 Saltstack release (in release candidate status at the time of this writing) so that we can manage our searches via orchestration.

Grafana dashboard orchestration using SaltStack

As mentioned in my post on Cloudwatch alarms, we (Lyft) believe that it should be easy to do the right thing and difficult to do the wrong thing. We operate on the concept “If you build it, you run it.” Running your own service isn’t easy, if you don’t have the right tools to help you, though.

We’re using Graphite and Grafana for time series data dashboards. With a consistent configuration management pattern all new services start with their data flowing into Graphite. Dashboard management is tricky, though. We encourage teams to add custom metrics to their services and use them in panels and rows for their services, but we also want to provide a number of consistent panels/rows for all services. We also want to avoid making teams go between multiple dashboards to monitor their own services.

SaltConf15: Sequentially Ordered Execution in SaltStack talk and slides

Here’s another talk that I gave at SaltConf15. It’s about sequentially ordered Salt and if you’ve read my blog posts on it, this probably won’t add a lot of technical info, but it’ll give a lot more context behind why you’d want to use Salt in a sequentially ordered way. Enjoy!

Sequentially Ordered Execution in SaltStack

Here’s the slides.. Note: though my blog is creative commons licensed, the slides are all rights reserved (sorry!).

SaltConf15: Masterless SaltStack at Scale talk and slides

I gave a talk at SaltConf15. It’s about masterless SaltStack, AWS orchestration, Docker management using Salt and other fun things. Enjoy!

Here’s the slides.. Note: though my blog is creative commons licensed, the slides are all rights reserved (sorry!).