Community-led development "The Apache Way"
This page will give you an overview of everything you always wanted to know about the Apache Software Foundation but were afraid to ask. The difference between membership and committership, who decides what, how elections take place, how is our infrastructure setup, what is the board, what is a PMC, what's the philosophy behind the incubator, and how the ASF is dealing with the incredible growth in new projects and contributors over the years. Come and see behind the scenes of the ASF.
The Apache Software Foundation (ASF) is a 501(c)3 non-profit public charity organization incorporated in the United States of America and was formed in 1999 primarily to:
provide a foundation for open, collaborative software development projects by supplying hardware, communication, and business infrastructure
create an independent legal entity to which companies and individuals can donate resources and be assured that those resources will be used for the public benefit
provide a means for individual volunteers to be sheltered from legal suits directed at the Foundation's projects
protect the 'Apache' brand, as applied to its software products, from being abused by other organizations
That's the dry fact, but how did all this come to be and what does it really mean in its details? We need to step back a little in history.
The foundation was created in 1999 by a group of people, that called themselves the "Apache Group" and had come together several years earlier, to continue to support and maintain the HTTPD web server written by the NCSA.
That server was freely available, came with source code and was licensed under a license that allowed very open modification and redistribution, but the original developers lost interest in that project and moved onto something else, leaving users with no support.
Some of those users started to exchange fixes (called "patches") and information on how to prevent problems and improve the existing software. Brian Behlendorf created a mailing list on his own machine for those users to collaborate to fix, maintain and improve that software.
The name 'Apache' was chosen from respect for the Native American Apache Nation, well-known for their superior skills in warfare strategy and their inexhaustible endurance. It also makes a cute pun on "a patchy web server" -- a server made from a series of patches -- but this was not its origin. The group of developers who released this new software soon started to call themselves the "Apache Group".
Between 1995 and 1999, the Apache HTTPD Web Server created by the Apache Group became the leader of the market (and currently still is, with more than 65% of the web sites in the world powered by it).
But as the web grew bigger, economical interests started to grow, and the Apache web site hosted new sister projects (such as the mod_ perl project, the PHP project, the Java Apache project). The need for a more coherent and structured organization that would shield individuals from potential legal attacks felt more and more necessary.
You can read more about ASF History.
Unlike other software development efforts done under an open source license, the Apache Web Server was not initiated by a single developer (for example, like the Linux Kernel, or the Perl/Python languages), but started as a diverse group of people that shared common interests and got to know each other by exchanging information, fixes and suggestions.
As the group started to develop their own version of the software, moving away from the NCSA version, more people were attracted and started to help out, first by sending little patches, or suggestions, or replying to email on the mail list, later by more important contributions.
When the group felt that the person had "earned" the merit to be part of the development community, they granted direct access to the code repository, thus increasing the group and increasing the ability of the group to develop the program, and to maintain and develop it more effectively.
We call this basic principle "meritocracy": literally, government by merit.
What is interesting to note is that the process scaled very well without creating friction, because unlike in other situations where power is a scarce and conservative resource, in the Apache group newcomers were seen as volunteers that wanted to help, rather than people that wanted to steal a position.
Being no conservative resource at stake (money, energy, time), the group was happy to have new people coming in and help, they were only filtering the people that they believed committed enough for the task and matched the human attitudes required to work well with others, especially in disagreement.
After explaining the structure of the ASF, we will see how the meritocracy relates to the various roles.
As the Apache Web Server started to grow in market share and popularity, due to synergy of its technical merit and to the openness of the community behind the project, people started to create satellite projects. Influenced by the spirit of the community they were used to, they adopted the same traditions of community management.
So, by the time the ASF was created, there were several separate communities, each focused on a different side of the "web serving" problem, but all united by a common set of goals and a respected set of cultural traditions in both etiquette and process.
These separate communities were referred to as "projects" and while similar, each of them exhibited little differences that made them special.
In order to reduce friction and allow for diversity to emerge, rather than forcing a monoculture from the top, the projects are designated the central decision-making organizations of the Apache world. Each project is delegated authority over development of its software, and is given a great deal of latitude in designing its own technical charter and its own governing rules.
At the same time, the cultural influence of the original Apache group was strong and the similarities between the various communities are evident, as we'll see later.
The foundation is governed by the following entities:
Board of Directors (board) governs the foundation and is composed of members.
Project Management Committees (PMC) govern the projects, and they are composed of committers. (Note that every member is, by definition, also a committer.)
Various Officers of the corporation, appointed by the board, who set Foundation-wide policies in specific areas (legal, brand, fundraising, etc.)
For all the details, read our Governance overview.
The board is responsible for management and oversight of the business and affairs of the corporation in accordance with the foundation Bylaws. This includes management of the corporate assets (funds, intellectual property, trademarks, and support equipment) and allocation of corporate resources to projects.
However, technical decision-making authority regarding the content and direction of the Apache projects is assigned to each respective project management committee.
The board is currently composed by nine individuals, elected between the members of the foundation. The bylaws don't specify the number of officers that the board should have, but historically, this was the number of the first board and it has never changed. The board is elected every year.
The board website has more information, the list of the current directors, schedule of meetings, and past minutes.
The Project Management Committees are established by resolution of the Board, to be responsible for the active management of one or more communities, which are also identified by resolution of the Board.
Each PMC consists of at least one officer of the ASF, who shall be designated chairperson, and may include one or more other members of the ASF.
The chair of the PMC is appointed by the Board and is an officer of the ASF (Vice President). The chair has primary responsibility to the Board, and has the power to establish rules and procedures for the day to day management of the communities for which the PMC is responsible, including the composition of the PMC itself. See further discussion about the role of PMC chair and why chairs are officers.
The role of the PMC from a Foundation perspective is oversight. The main role of the PMC is not code and not coding - but to ensure that all legal issues are addressed, that procedure is followed, and that each and every release is the product of the community as a whole. That is key to our litigation protection mechanisms.
Secondly the role of the PMC is to further the long term development and health of the community as a whole, and to ensure that balanced and wide scale peer review and collaboration does happen. Within the ASF we worry about any community which centers around a few individuals who are working virtually uncontested. We believe that this is detrimental to quality, stability, and robustness of both code and long term social structures.
We firmly believe in hats. Your role at the ASF is one assigned to you personally, and is bestowed on you by your peers. It is not tied to your job or current employer or company.
However those on the PMC are kept to a higher standard. As the PMC, and the chair in particular, are eyes and ears of the ASF Board, it is you that we rely on and need to trust to provide legal oversight.
The board has the faculty to terminate a PMC at any time by resolution.
The meritocracy typically has various roles within each individual Apache project communities:
A user is someone that uses our software. They contribute to the Apache projects by providing feedback to developers in the form of bug reports and feature suggestions. Users participate in the Apache community by helping other users on mailing lists and user support forums.
A developer is a user who contributes to a project in the form of code or documentation. They take extra steps to participate in a project, are active on the developer mailing list, participate in discussions, provide patches, documentation, suggestions, and criticism. Developers are also known as contributors .
A committer is a developer that was given write access to the code repository and has a signed Contributor License Agreement (CLA) on file. They have an apache.org mail address. Not needing to depend on other people for the patches, they are actually making short-term decisions for the project. The PMC can (even tacitly) agree and approve it into permanency, or they can reject it. Remember that the PMC makes the decisions, not the individual committers.
A PMC member is a developer or a committer that was elected due to merit for the evolution of the project and demonstration of commitment. They have write access to the code repository, an apache.org mail address, the right to vote for the community-related decisions and the right to propose an active user for committership. The PMC as a whole is the entity that controls the project, nobody else. In particular, the PMC must vote on any formal release of their project's software products.
The Chair of a Project Management Committee (PMC) is appointed by the Board from the PMC Members. The PMC as a whole is the entity that controls and leads the project. The Chair is the interface between the Board and the Project. PMC Chairs have specific duties.
An ASF member is a person who was nominated by current members and elected due to merit for the evolution and progress of the foundation. Members care for the ASF itself. This is usually demonstrated through the roots of project-related and cross-project activities. Legally, a member is a "shareholder" of the foundation, one of the owners. They have the right to elect the board, to stand as a candidate for the board election and to propose a committer for membership. They also have the right to propose a new project for incubation (we'll see later what this means). The members coordinate their activities through their mailing list and through their annual meeting. We have a full listing of Apache Members.
The Apache projects are managed using a collaborative, consensus-based process. We do not have a hierarchical structure. Rather, different groups of contributors have different rights and responsibilities in the organization.
Since the appointed Project Management Committees have the power to create their own self-governing rules, there is no single vision on how PMCs should run a project and the communities they host.
At the same time, while there are some differences, there are a number of similarities shared by all the projects:
Communication is done via mailing lists. These identify "virtual meeting rooms" where conversations happen asynchronously, which is a general requirement for groups that are so geographically distributed to cover all time zones (like it's normally the case for the various Apache communities).
Some projects additionally use more synchronous messaging (for example, IRC or instant messaging). Voice communication is extremely rare, normally because of costs and the language barrier (speech is harder to understand than written text).
In general, asynchronous communication is much more important because it allows archives to be created and it's more tolerant on the volunteer nature of the various communities.
Projects are normally auto governing and driven by the people who volunteer for the job. This is sometimes referred to as "do-ocracy" -- power of those who do. This functions well for most cases.
When coordination is required, decisions are taken with a lazy consensus approach: a few positive votes with no negative vote is enough to get going.
Voting is done with numbers:
+1 -- a positive vote
0 -- abstain, have no opinion
-1 -- a negative vote
The rules require that a negative vote includes an alternative proposal or a detailed explanation of the reasons for the negative vote.
The community then tries to gather consensus on an alternative proposal that resolves the issue. In the great majority of cases, the concerns leading to the negative vote can be addressed.
This process is called "consensus gathering" and we consider it a very important indication of a healthy community.
Specific cases have some more detailed voting rules.
While there is not an official list, these six principles have been cited as the core beliefs of philosophy behind the foundation, which is normally referred to as "The Apache Way":
collaborative software development
commercial-friendly standard license
consistently high quality software
respectful, honest, technical-based interaction
faithful implementation of standards
security as a mandatory feature
All of the ASF projects share these principles. Similarly, Apache projects are required to govern themselves independently of undue commercial influence.
All projects are composed of volunteers and nobody (not even members or officers) are paid directly by the foundation for their job. There are many examples of committers that are paid to work on the projects, but never by the foundation themselves, but rather by companies or institutions that use the software and want to enhance it or maintain it.
Note that the ASF does contract out various services, including accounting, Press and Media relations, and infrastructure system administration.
All of the ASF including the board, the other officers, the committers, and the members, are participating as individuals. That is one strength of the ASF, affiliations do not cloud the personal contributions.
Unless they specifically state otherwise, whatever they post on any mailing list is done as themselves. It is the individual point-of-view, wearing their personal hat and not as a mouthpiece for whatever company happens to be signing their paychecks right now, and not even as a director of the ASF.
All of those ASF people implicitly have multiple hats, especially the Board, the other officers, and the PMC chairs. They sometimes need to talk about a matter of policy, so to avoid appearing to be expressing a personal opinion, they will state that they are talking in their special capacity. However, most of the time this is not necessary, personal opinions work well.
Some people declare their hats by using a special footer to their email, others enclose their statements in special quotation marks, others use their apache.org email address when otherwise they would use their personal one. This latter method is not reliable, as many people use their apache.org address all of the time.
We endeavour to conduct as much discussion in public as possible. This encourages openness, provides a public record, and stimulates the broader community.
However sometimes internal private mail lists are necessary. You must never divulge such information in public without the express permission of the list. Also never copy an email between private and public lists (no Cc). Such an event would go beyond the normal need for email ettiquette and be a serious breach of confidence. It could have serious ramifications, cause unnecessary confusion and ill-informed discussion.
Private lists are typically only used for matters pertaining to people as individuals (like voting in new committers), and legal matters that require confidentiality.
In order for new projects to be created, the ASF created a project called Incubator which is responsible to help new efforts to join the foundation.
Since the meritocratic rules operate across the ASF from bottom to top, it is vital for the long-term stability of such a form of government, that the initial set of committers has to know very well the dynamics of such a system, as well as share the same philosophical attitude toward collaboration and openness that the ASF expects from its projects.
The incubator is responsible for:
filtering the proposals about the creation of a new project or sub-project
help the creation of the project and the infrastructure that it needs to operate
supervise and mentor the incubated community in order for them to reach an open meritocratic environment
evaluate the maturity of the incubated project, either promoting it to official project/ sub-project status or by retiring it, in case of failure.
It must be noted that the incubator (just like the board) does not perform filtering on the basis of technical issues. This is because the foundation respects and suggests variety of technical approaches. It doesn't fear innovation or even internal confrontation between projects which overlap in functionality.
The incubator filters projects on the basis of the likeliness of them becoming successful meritocratic communities. The basic requirements for incubation are:
a working codebase -- over the years and after several failures, the foundation came to understand that without an initial working codebase, it is generally hard to bootstrap a community. This is because merit is not well recognized by developers without a working codebase. Also, the friction that is developed during the initial design stage is likely to fragment the community.
the intention to assign sufficient intellectual property rights to the software to the ASF to the foundation -- this allows the foundation to obtain an irrevocable and permanent right to redistribute and work on the code, without fearing lock-in for itself or for its users, while still allowing the original author to maintain their copyright.
a sponsoring ASF member or officer -- this person will act as the main mentor, giving directions to the project, helping out in the day-to-day details and keeping contact with the incubator PMC.
The incubation period normally serves to estimate whether or not:
It might seem rather easy to achieve, but it must be remembered that in a volunteer and highly selective environment, attracting new committers is not automatic.
Diversity of committership is important for two main reasons:
it gives long term stability to the project development: in fact, with all the developers affiliated to the same entity, the chance of seeing all of them moving away from the project at the same time is much greater than with a community of individuals affiliated to unrelated entities.
it gives a greater variety of technical visions: something that guarantees a better adherence to environment and user's needs, thus a higher change of finding real-life use of the software.
Along with the Incubator, the foundation has several other cross-foundation projects. For example the ASF does not have offices or buildings, it's a virtual entity that exists only on the internet and the technical infrastructure that enables it to operate is managed by the Infrastructure team.
These and other cross-foundation projects are described on the Foundation Projects page.
The ASF also hosts some foundation-wide mailing lists, which are detailed on the Mailing Lists page.
Within the first 10 years of operation, the ASF represents one of the best examples of an open organization that has found balance between structure and flexibility. We have grown from 200 committers to around 3000, and that number continues to grow on a daily basis. We have been able to create several software products that are leaders in their market. We have also been able to find balance between openness and economical feasibility. This has earned us respect from a range of people, from single individuals to multinational corporations. We hope to continue to provide inspiration for businesses, governments, education, and for other software foundations.