Intro to Platform Engineering

What is Platform Engineering

Platform engineering is an approach to building and managing technology platforms within an organization. A platform provides a common foundation that software products and services that can be built on top of. Platform engineers focus on creating tools, services, and infrastructure that serve as the core technology other engineers use to develop applications and software.

Key Responsibilities of Platform Engineers

Some key responsibilities of platform engineers on a platform team include:

• Developing shared services and APIs that can be reused across multiple products/teams
• Creating developer tools and systems to improve engineering productivity
• Building infrastructure and automation to simplify deployments and operations
• Establishing patterns, standards, and best practices for how software gets built and run
• Providing self-service access to technology resources for engineers
• Managing and monitoring platform systems for reliability, scalability, and security

Platform engineering focuses on internal customers (engineers within the same company) as opposed to external customers. The goal is to abstract away common complexity and provide configurable building blocks that engineers can use to deliver products and features faster. Well-designed platforms allow engineers to focus on differentiating business logic rather than reinventing foundational technology.

Why Platform Engineering Matters

Platform engineering and having an internal developer platform strategy are becoming increasingly important for software engineering organizations building software products and applications. Here are some of the key benefits of investing in platform engineering capabilities:

Accelerate Product Delivery

Platform teams can significantly accelerate feature development by providing reusable components and services for development teams. Engineering teams don't have to rebuild foundational technology for every product and can focus on their specific problem domain. A platform removes friction and burden from product teams and lets them concentrate on innovation and customer value.

Drive Engineering Efficiency

Platforms enable engineers to be more productive by abstracting away infrastructure complexity and providing self-service access to resources. Teams spend less time on mundane tasks like managing servers or configuring monitoring and can dedicate more cycles to advancing the product.

Promote Best Practices

An effective platform promotes standards that drive quality and consistency across engineering projects. This includes establishing common design patterns, defining project architecture principles, standardizing coding practices, and enforcing security policies. The result is more reliable and robust applications.

Facilitate Collaboration

A platform provides a catalog of shared services that multiple teams can discover and leverage. This increased interdependency fosters better communication and collaboration across products and groups. The platform creates synergies across the engineering organization.

Recruit and Retain Talent

Engineers want to focus on solving interesting problems and building products, not infrastructure. A company that invests in its platform and engineer experience attracts top talent and enables the organization to produce better work.

Achieve Economies of Scale

There are major cost efficiencies in reusing platform infrastructure across engineering teams versus fragmented systems built redundantly. Companies can achieve economies of scale from both a cost and productivity perspective.

Elements of a Strong Platform

Robust developer platforms involve multiple components working together to maximize productivity. Here are some key elements:

Developer Tools

High-quality developer tools boost efficiency. This includes IDEs, CI/CD, observability, source control, and workflow systems like JIRA. Self-service access makes onboarding frictionless.

Shared Services

Reusable software services provide common functionality via APIs that teams can leverage:

• Data and storage - Databases, object storage, caching, search, stream processing
• Messaging - Async task queues, message buses, notifications
• Security - Authentication, authorization, secrets management
• Serverless - Functions like AWS Lambda to run isolated code
• Machine learning - Shared models, prediction services
• Media processing - Transcoding, thumbnailing, watermarking

Foundational Building Blocks

Core platform foundations that provide baseline functionality:

• Data - Data stores, pipelines, lineage, governance
• Infrastructure - Networking, runtimes, containers, resource management
• Observability - Logging, tracing, metrics, alerting
• Security - Encryption, certificates, secrets, RBAC
• Testing - Test automation frameworks, mocks, performance

Internal Developer Experience

Optimizing the experience for internal engineers:

• Self-service - On-demand provisioning of resources
• Developer portal - Documentation, guides, API reference
• Community - Communication channels, mailing lists, chat
• Onboarding - Developer training, support, office hours
• Request management - User feedback, feature requests, roadmap input
• Status transparency - System dashboards, incident reports

Challenges in Platform Engineering

Building a robust internal developer platform presents challenges including:

Adoption - Getting teams to leverage shared services vs going rogue. Must demonstrate value.

Funding - Leadership must see the long-term benefits of platform investment.

Fragmentation - Siloed teams duplicating effort without coordination.

Communication - Facilitating cross-team discussions and gathering feedback.

Talent - Platform skills span software engineering, ops, and product. Difficult combination.

Security and compliance - Added risks and responsibilities inherited from the services built atop the platform.

Ownership - Should the platform be owned centrally or by individual teams?

Complexity - Managing complexity as platforms scale. Avoiding technical debt.

Evolution - Incremental changes while maintaining legacy support. Avoiding big bang rewrites.

Focus - Balancing custom platform services vs leveraging managed cloud services.

Best Practices for Platform Success

Here are some best practices that can lead to effective platform implementations:

• Start small - Deliver minimum viable capabilities, expand incrementally
• Developer experience - Should make engineers' lives easier, not harder
• Collaboration - Close communication between platform and product teams
• Customer obsession - Solve internal engineers' pain points, don't chase fads
• Leverage cloud services - Don't reinvent the wheel unnecessarily
• Automate relentlessly - Manual processes don't scale
• Standardize judiciously - Some standards enable agility long term
• Central control vs autonomy - Balance governance with developer freedom
• Observability first - Logging, metrics, and tracing pave the path
• Loosely coupled architecture - Isolate services for independent evolution
• Plan for growth - Build with expected future scale in mind

Organizational Models

There are a few common organizational structures for platform teams:

Centralized Platform Team

A dedicated cross-functional team owns the core developer platform from top to bottom. They provide services to other engineering teams but don't build end-user features.

Embedded Platform Engineers

Platform engineers integrate directly into product feature teams rather than a separate central team. Still, coordinate standards across the organization.

Hybrid Approach

A central platform team governs the overall strategy and foundational services while embedded engineers fill gaps on specific teams as needed.

Developer Productivity Teams

Some engineering organizations have teams entirely focused on improving developer productivity, tooling, and infrastructure.

Roles on Platform Teams

Here are typical roles that contribute to platform teams:

• Software Engineers - Build platform components across the stack from frontend to infrastructure.
• Product Managers - Gather requirements, define roadmaps, and set priorities for platform capabilities.
• Developer Advocates - Promote platform adoption, support onboarding, and foster community.
• Solutions Architects - Establish high-level architecture strategy and guide technology choices.
• SRE/DevOps Engineers - Deploy, operationalize, and monitor services. Ensure reliability and uptime.
• Data Engineers - Build pipelines and infrastructure for analytics, reporting, and machine learning.
• Security Engineers - Provide security guidance, enforce policies, and ensure platform integrity.
• QA Engineers - Automated testing, release validation, defect management.

While titles vary, platform teams need a mix of software engineering, operations, and product management skill sets.

Measuring Success

Important metrics to track how well a platform team is achieving its goals:

• Adoption - % of teams using platform services, growth in usage over time
• Satisfaction - Developer experience ratings collected via surveys, interviews
• Productivity - Improved team output or velocity since the platform rollout
• Reliability - Change failure rate, service uptime, performance
• Time to market - Reduction in project kickoff to launch due to platform leverage
• Cost efficiency - Decrease in redundant spending on infrastructure, tooling

The Platform Engineering Process

Here are some best practices for approaching platform builds:

Understand Internal Customers

Get insight into engineers' pain points through techniques like value stream mapping and jobs-to-be-done. Empathize with their realities.

Start with High Leverage Problems

Identify the areas that will have the biggest impact on developer velocity and focus on solving those first.

Design with Developer Experience in Mind

APIs, SDKs, and tooling should be intuitive and frictionless for engineers. If not, they won't adopt them.

Maintain Close Communication

Keep a feedback loop between platform teams and the consumers of platform services to continually align.

Plan for Evolution

Build in flexibility for changing requirements and support for legacy consumers over time.

Invest in Automation and Observability

Manual processes don't scale. Logging, metrics, and tracing pave the path to reliability.

Embrace Compliance as Code

Make compliance checks part of CI/CD pipelines to bake security in from the start.

Build Reusable Abstractions

Design components that abstract complexity away from consumers while remaining customizable under the hood.

Monitor Ecosystems

Stay up to date on new technologies, standards, and practices to inform platform roadmap priorities.

The Emergence of Platform Engineering Teams

As software engineering organizations grow, the need emerges for an internal team dedicated to maximizing developer productivity. Reasons platform teams arise:

• Engineering teams start spending too much time on undifferentiated infrastructure versus core product work.
• Duplicated effort across teams from not sharing common solutions.
• Difficulty onboarding and ramping up new engineers without standardized tooling and stacks.
• Unreliable, slow legacy systems hinder developer velocity. Lack of automation, testing, and monitoring.
• Security and compliance risks increase with fragmented systems and inconsistent practices.
• Developer experience suffers when technical roadblocks impede product innovation.

The platform team addresses these issues by providing reusable building blocks that reduce friction. They aim to remove distractions so engineers can focus on delivering differentiated value. Mature engineering organizations view platform investment as a force multiplier that pays dividends over time.

The Future of Platform Engineering

As technology continues evolving rapidly, platform engineering will only increase in importance for software organizations. Here are some key trends shaping the future landscape:

• Everything as Code - Platform tooling will enable full environments to be defined and managed as code.
• Increased Adoption of Cloud Services - More reuse of managed cloud infrastructure vs custom platform builds.
• Rise of Internal Developer Portals - Self-service dev portals for documentation, sandboxes, stacks, and tooling.
• Focus on Security and Compliance - Greater need for security automation, policy enforcement, and audit trails.
• Value Stream Optimization - Platforms will facilitate easier tracking and optimization of engineering value streams.
• Mainstreaming of Site Reliability Principles - SRE skills expected from all platform engineers.
• Low-Code Development - Platforms will empower low-code solutions and citizen development where optimal.
• Everything Gets Instrumented - Increased telemetry across all systems, apps, and processes.

To attract top talent and accelerate innovation, investing in internal developer platforms will be key for ambitious engineering organizations. Platform capabilities provide the foundation for engineering teams to thrive and maximize their productivity over the long term.

Key Takeaways

• Platform engineering focuses on enabling engineering productivity through reusable services and infrastructure.
• Platform teams aim to remove the friction that slows down product teams so they can focus on innovation.
• Effective platforms provide self-service access to data, security, CI/CD, ML models, and other utilities via APIs.
• Platform capabilities should optimize for internal developer experience.
• Organizations have flexibility in how they structure platform teams based on their needs and culture.
• Platform engineering requires a combination of technical and product management skills to understand internal customer needs.
• Investing in platform engineering and internal developer platforms provides a competitive advantage by accelerating developer velocity across the engineering organization.