Site Reliability Engineer, Reliability: Foundations
The GitLab DevSecOps platform empowers 100,000+ organizations to deliver software faster and more efficiently. We are one of the world’s largest all-remote companies with 2,000+ team members and values that foster a culture where people embrace the belief that everyone can contribute. Learn more about Life at GitLab.
Site Reliability Engineers (SREs) are responsible for keeping all user-facing services and other GitLab.com production systems running smoothly. SREs are a blend of pragmatic operators and software craftspeople that apply sound engineering principles, operational discipline, and mature automation to our environments and the GitLab codebase. We specialize in systems, whether it be networking, the Linux kernel, or some more specific interest in scaling, algorithms, or distributed systems.
The Foundations Team owns the reliability, scalability, and security of the existing core infrastructure of GitLab.com. We seek to reduce the effort required to provide our core infrastructure services, and to enable other teams to self-serve core infrastructure that allows them to more efficiently/effectively run their services. We work on some of the most foundational, critical, and visible parts of the infrastructure supporting .com, through collaboration, technical excellence, continuous learning, and balancing courage with making changes as safely as we can. All while aiming for a sense of fun and connection within the team.
GitLab.com is a unique site and it brings unique challenges–it’s the biggest GitLab instance in existence. In fact, it’s one of the largest single-tenancy open-source SaaS sites on the internet. The experience of our team feeds back into other engineering groups within the company, as well as to GitLab customers running self-managed installations
- Automating every operational task is a core requirement for this role. For example, package updates, configuration changes across all environments, creating tools for automatic provisioning of user facing services, etc.
- Responding to platform emergencies, alerts, and escalations from Customer Support.
- Managing software life-cycles (e.g. Operating Systems) with a minimum of manual effort.
- Managing and improving our edge tech stack to ensure safe and efficient routing.
- Planning for new service roll-outs, expansion and capacity management of existing services, and working with other teams to optimize their resource consumption.
As an SRE you will:
- Be on a PagerDuty rotation to respond to GitLab.com availability incidents and provide support for service engineers with customer incidents.
- Analyze existing, and create and maintain new GitLab.com Service Level Objectives.
- Troubleshoot, evaluate and resolve operational challenges contributing to defined SLO's.
- Define, improve, and engage in adapting architectural application bottlenecks as observed on GitLab.com.
- Work with other engineering stakeholders on resolving larger architectural bottlenecks and participate by offering GitLab.com point of view.
- Work in close collaboration with software development teams to shape the future roadmap and establish strong operational readiness across teams.
- Scale systems through automation, improving change velocity and reliability.
- Leverage technical skills to partner with team members and be comfortable diving into a problem as needed.
- Work with counterparts in other teams of the Infrastructure department to improve infrastructure running with Chef, Terraform and Kubernetes.
- Ensure that monitoring and alerting alert on symptoms and not on outages.
- Document every action so your findings turn into repeatable actions–and then into automation.
- Debug production issues across services and levels of the stack.
You may be a fit to this role if you:
- Have extensive experience with Cloud-native Engineering, including a proficiency in Kubernetes and the associated ecosystem of running cloud-native services
- Have experience with Infrastructure as Code, including a proficiency in Chef and Terraform
- Have a deep understanding of network concepts and experience with our Edge Stack (Cloudflare, HAProxy, Cloud VPCs, Rate limiting, GCP IAM permissioning, etc)
- Are able to reason about large systems - how they work on large scale, edge cases, failure modes, behaviors.
- Know your way around Linux and the Unix Shell.
- Have experience in collaborating and communicating asynchronously.
- Have an urge to document all the things so you don't need to learn the same thing twice.
- Have an enthusiastic, proactive attitude. When you see something broken, you can't help but fix it.
- Have a strong sense for action and know how to iterate through a problem quickly.
- When faced with vague requirements, you keep asking questions until there is clarity.
- Share our values (both GitLab values and the Foundations values), and work in accordance with those values.
- Are able to leverage GitLab as your day to day go-to tool.
- Bonus: Strong programming skills as a (former) software engineer - Preferably with Ruby and/or Go.
Projects you could work on:
- Work from the Foundations team’s issue tracker, driving changes required to reliably support GitLab at GitLab.com scale.
- Work on improving automation and usability of the services Foundations owns.
- Work on projects that create deployment packages of GitLab such as, GitLab Helm Chart, omnibus-gitlab, etc.
- Code infrastructure automation with Chef and Terraform.
- Improve the reliability and security of our services through upgrading technologies and addressing tech debt, for example by implementing Vault for secrets management.
Senior Site Reliability Engineer Criteria
- Deep knowledge in 2 areas of expertise and general knowledge of all areas of expertise. Capable of mentoring Junior in all areas and other SRE in their area of deep knowledge.
- Contributes small improvements to the GitLab codebase to resolve issues
- Identifies significant projects that result in substantial cost savings or revenue
- Identifies changes for the product architecture from the reliability, performance and availability perspective with a data driven approach.
- Proactively works on the efficiency and capacity planning to set clear requirements and reduce the system resources usage to make GitLab cheaper to run for all our customers.
- Identifies parts of the system that do not scale, provides immediate palliative measures and drives long term resolution of these incidents.
- Identifies Service Level Indicators (SLIs) that will align the team to meet the availability and latency objectives.
Collaboration and Communication:
- Knows a domain really well and radiates that knowledge
- Performs and runs blameless RCAs on incidents and outages, aggressively looking for answers that will prevent the incident from ever happening again.
Influence and Maturity:
- Leads by example, both in terms of technical execution and collaboration and communication standards.
- Shows ownership of a major part of the infrastructure.
- Trusted to de-escalate conflicts inside the team
Site Reliability Engineers have the following job-family performance indicators:
- GitLab.com Availability
- GitLab.com Performance
- Apdex and Error SLO per Service
- Mean Time to Detection
- Mean Time to Resolution
- Mean Time Between Failure
- Mean Time to Production
- Disaster Recovery Time to Recovery
Country Hiring Guidelines: GitLab hires new team members in countries around the world. All of our roles are remote, however some roles may carry specific location-based eligibility requirements. Our Talent Acquisition team can help answer any questions about location after starting the recruiting process.
GitLab is proud to be an equal opportunity workplace and is an affirmative action employer. GitLab’s policies and practices relating to recruitment, employment, career development and advancement, promotion, and retirement are based solely on merit, regardless of race, color, religion, ancestry, sex (including pregnancy, lactation, sexual orientation, gender identity, or gender expression), national origin, age, citizenship, marital status, mental or physical disability, genetic information (including family medical history), discharge status from the military, protected veteran status (which includes disabled veterans, recently separated veterans, active duty wartime or campaign badge veterans, and Armed Forces service medal veterans), or any other basis protected by law. GitLab will not tolerate discrimination or harassment based on any of these characteristics. See also GitLab’s EEO Policy and EEO is the Law. If you have a disability or special need that requires accommodation, please let us know during the recruiting process.
Something looks off?