The Infrastructure Team

The ASF Infrastructure team provides and manages all infrastructure and services for the Apache Software Foundation, and for each project at the Foundation. This infrastructure includes the various machines and their operating systems, the mailing lists, the version control systems, committer accounts, the distribution mirroring system, a variety of issue tracking systems, and more. The ASF Infrastructure team is responsible for systems administration, and as well as supporting existing projects at The Apache Software Foundation, the ASF Infrastructure team works hard to provide services and setup for each new project that joins the Foundation via the Apache Incubator.

The ASF Infrastructure team is made up of ASF committers, some of whom are paid sysadmin contractors. Like every ASF project, the non-contracted committers work as volunteers - and we could always do with more help! Those volunteers work alongside the contractors in support of decisions approved, either implictly or explictly, by the Infrastructure President's Committee. Decision-making follows a distributed hierarchy instead of the typical fully group-centric decision-making of an Apache project, see below for details on how this works in practice.

Getting started

If you're a willing volunteer, who can tell the difference between a server and a hifi amp, this is for you! There are plenty of opportunities for you to help out with the small tasks that need to be done every day to keep the ASF Infrastructure alive. Some are self-explanatory, and you'll learn more about the rest of them as you get involved.

There are some basic things you should be able to do, if you're going to help out with the ASF Infrastructure team. You probably do them all already, in other ASF projects.

  • Read infrastructure mailing lists and be aware of what is going on. Read #asfinfra as well.

  • Work on documentation, especially procedural documentation and answers to FAQs. It is more effective if we can simply refer people to a webpage.

  • Answer questions that other people ask on infrastructure@. This is immensely beneficial to the rest of the team, as it allows them space to get on with other things.

  • Work through the Issue Tracker and submit patches, comments, links, etc, that may help in resolving them.

  • Look for support requests on infrastructure@ as they come in and volunteer to handle them.

  • Add entries to the Issue Tracker for issues raised via e-mail that are not yet being handled. We don't want them slipping through the cracks.

  • Work on in-house tasks. At any time there are several outstanding TODOs involving the in-house services, patches, and scripts; not everything is recorded on the issue tracker.

Care and feeding of your sysadmin

If you're going to help out on the ASF Infrastructure team, you'll come into contact with real, live sysadmins. It can take a little while to get used to these creatures, but it's well worth making the effort. There are some definite first-steps to take, which are outlined here. For further information, talk to more experienced team members - or just watch and learn!

Some tasks on the Infrastructure team will require you to use SVN. Your sysadmin shouldn't need to tell you how to do this - there's plenty of information online. You'll also need to be able to use Jira. This might be slightly trickier - but remember to search the web before you pester your sysadmin. We don't have great documentation on which method to use when - feel free to ask your sysadmin on a case-by-case basis.

Being part of the Infrastructure team does mean a lot of email. If you're having trouble, figure out how to sort and filter your email before subscribing to many more email lists - this will make many aspects of your life easier, not just your work at the ASF or on the Infrastructure team! Make sure that you quote other people's emails correctly when replying, and always use descriptive and informative subject lines.

Do as much preparation work as you can for any task that you take on, to minimize the amount of time your sysadmin needs to spend on it. Get familiar with the contents of the www.apache.org/dev/ website. Don't expect to join the ranks of the sysadmins just because you do this in your day job - like every project at the ASF, you need to start small, submit good patches, and build up trust and merit.

If you ask a question to which the answer is not in the documentation, please submit a documentation patch. If you know the answer to a question, please always answer it even if you are not otherwise an active Infra participant.

If a service is down, discussing it on #asfinfra is probably a better idea than sending an e-mail about it, since usually the Infra team is already aware of that anyway and working on it (we have monitoring systems and alerts). Especially if there seems to be an issue with e-mail, don't send e-mail about it (you would be surprised how many people do). Instead, jump on Hipchat and see if you can help.

How to get on well with the Infrastructure team

  • Hang out for a while until you become familiar with processes and people.

  • Thoroughly research an issue when you request a change, and provide the results of your research in an easy to digest, easy to verify format. Don't expect people to take your word for things, but provide relevant links. Provide sample commandline statements you think may work for resolving a particular task if possible. At a minimum, always RTFM!

  • Be conservative in sending e-mails. Keep e-mails concise and to the point. Send as few e-mails as possible.

  • Say "thank you" every now and then, but don't spend an entire e-mail on that.

  • Don't complain.

  • Be patient, very patient. Certain requests may be handled in a day, others may take a week or two, more if it's a complex issue, or indefinitely if you didn't RTFM or didn't research your request properly.

  • Don't volunteer for stuff, and then not finish it. If you have troubles then always ask for help.

  • Don't just come here to look after your pet project. We need people to attend to issues across the whole ASF.

We really do need you

If you follow the Infrastructure mailing lists for a while, you'll soon start to see the areas that need attention, and you'll become aware of what kind of attention they need. You can then volunteer to give that attention, or at least write up a problem description and potential solution process, and file that in the Issue Tracker.

Most of the Infrastructure tasks don't require root access - and while they may not sound very glamorous, it is very, very neccessary that these things are done. If you can help out with these things, you will earn the respect and appreciation of the team - and the huge appreciation of all the people who rely on the ASF Infrastructure every day, to develop and use the many ASF projects that live thereon.

Decision making

Decision making works a little differently for the ASF Infrastructure team than you might be used to from the average ASF software development project.

Sometimes we vote, sometimes people just do things because they know they make sense. For small things, "Just F'ing Do It" makes sense; for larger things, a discussion (recorded on a mailing list) is warranted. Always keep the team in the loop, even if you JFDI a change.

Some people are sufficiently familiar with the configuration of particular machines, and the operation of the Infrastructure team, to have the authority to "lay down the law". More people may have sufficient privileges to change stuff, but will only do so after extensive consultation. The best way to learn how we make decisions is to hang around and see.

A sound, technical argument will always be heard out, even if it's not ultimately accepted. Rules and points of order will generally be disregarded. This is the only way Infra can stay productive.

More information and resources

The Infrastructure team is a "President's Committee" rather than a "project". David Nalley chairs the committee at the moment (meaning he approves regular reports to the ASF President every month). The committee consists of people listed here and the definition for who is "on the infrastructure team" is here. They are all ASF committers.

Mailing lists

There are various infrastructure mailing lists. See notes about their purpose and who can subscribe. As a new volunteer, you will be mainly interested in "infrastructure@" which is the main list for reporting issues. Please help by answering the simple questions, like directing people to specific documentation. Other infra lists are for developing solutions and doing core operations.

Hipchat

Infra maintains a publicly accessible Hipchat channel that is used for realtime issues and is especially useful when the mail server is down. It is open to anyone who subscribes to the infra mailing lists. We know who you are.

Emergency notifications

This should be used with care, and requires infrastructure karma to carry out. See the emergency contact page for more information.

Generally, you will want to try to ping an infrastructure representative on the Hipchat first, and if all else fails, you can resort to the SMS notification if need be, but, to reiterate, use this feature with care - waking root up in the middle of the night because your buildbot fails is not acceptable use of this feature.

Website

There is a website for infrastructure information at www.apache.org/dev/ which also includes various information to assist all committers and developers. See Updating the Infrastructure web site.

Version control system

There is a Subversion repository https://svn.apache.org/repos/infra/infrastructure which is only available to people on the infrastructure team. Some of its subdirectories have more relaxed access restrictions. Talk to us if you need access.

There is also a Git repository https://git-wip-us.apache.org/repos/infra/asfgit-admin.git which is also only available to people on the infrastructure team. Talk to us if you need access.

Issue tracker

There is a project in Jira for managing issues, called INFRA. Anyone can submit issues. There is a custom policy in place where some issues are marked as more secure.