Kubernetes Infrastructure Project

Developer Empowered Operations

"We Build it, we deploy it, we manage and maintain high availability & quality of service in production."

Stages of Maturity:


The key benefits of standardizing patterns and technology:

  • Faster delivery velocity
  • More flexibility for developers to work on different applications, services or components
  • Reduced surface area for security vulnerabilities.
  • Fewer moving parts to maintain, upgrade and learn.

RFC: FSDI Technology Initiative

Request for comments:

i. This is a joint Developer, QA & Operations initiative by Ardeshir, Kevin, Dan, Ryan and Vitaly. We appreciate comments and suggestions.

ii. Please let us know if you have any concerns and recommendations. (email us @DevOps)

iii. We believe without Developer, QA, and the broader engineering teams’ support, this initiative could not move forward, so your participation is paramount from the start, we look forward to adding your comments, suggestions, concerns and names to this initiative.

The objectives and goals for the Kubernetes Infrastructure Project:

The FSDI technology operations team believes there are significant gains to standing up our own Kubernetes Cluster. Kubernetes is an open source, cloud agnostic system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications. The open source project is hosted by the Cloud Native Computing Foundation

  • The main benefit , that we can gain is significant reduction of infrastructure costs for non-prod environments, by using cheaper compute instances and more precise resource allocation.
  • Secondary benefit is having less vendor lock-in, allowing some of the apps to flow freely between Azure and AWS, or use one as a failover for the other.
  • Third is having a standardized environment framework for each new and existing application, while allowing dev teams their preferred tools freely. (They can run whatever distribution, CI tool or monitoring system they wish, as long as it is defined to run inside of the Cluster.)

Short term gains:

  1. High-available cluster running on a mix of on-demand and spot-instances on AWS.
  2. Billing system in place that takes AWS Billing info for the resources that our cluster is utilizing, correlates it to object metrics we collected through the K8S metrics API or other means, and then splits the bill accordingly for each service/project/namespace + idle resources.
  3. Pre-cooked services like common database engines and dev tools (MySQL, Jenkins, MongoDB etc.), easily available and deployable by dev teams
  4. Guidelines, documentation, templates, examples etc. available for dev teams to have an easier way to start or migrate to k8s
  5. A workflow set for Cloud SRE support requests. A Jira board and slack channel.
  6. An Update workflow set for compute resources upgrades and K8S elements updates that minimizes impact on running services.

Long term goals:

  1. Cloud Agnostic, Cross-Cloud Cluster with both Windows and Linux containers.
  2. Expand billing system to include Azure.
  3. Minimize the "idle" bill portion
  4. Expand K8S usage to prod environments

Drawbacks:

  • Complicated configuration: Kubernetes consists of many atomic services (API Server, etcd, Proxy) which must be securely integrated.
  • Infrastructure maintenance. That includes both to K8S elements and cloud infrastructure.
  • Complex migrating strategies. Teams with no docker/container experience need to overcome a slight learning curve, in order to gain the benefits of running their applications on a Kubernetes cluster.

Contributors:

Vitaly, Kevin, Dan, Ryan, Ardeshir