Methodology

Introduction

In the last twenty years, the nonprofit sector has evolved far beyond charity work to encompass myriad organizations, coalitions, movements, and even businesses that serve a diversity of objectives and engage in many ways with public and private institutions. NGO ADVISOR monitors the international nonprofit community for ideas, practices, and models that challenge traditional approaches to the non-governmental organization, or NGO. An essential monitoring tool is our list of the Top 500 NGOs, a concept initially developed by the former editor-in-chief of The Global Journal as a Top 100 NGOs list. In this iteration, we have expanded our ranking to include 500 organizations, entirely re-think our scoring methodology, and moved to a new platform at ngoadvisor.net.

NGO ADVISOR is currently chaired by Jean-Christophe Nothias, former editor-in-chief of The Global Journal, who brings years of expertise in journalism, policy, and development to the table. NGO ADVISOR publishes the Top 500 NGOs to serve as a lens to bring the transformations of the NGO sector into focus and to magnify the evolving range of nonprofit values. Using a journalistic approach to research and reporting, our rankings represent the human aspects of NGO activity that are rarely captured by stock indices or financial statements. NGO ADVISOR’s overall goal is to engage everyone interested in the power of the nonprofit, whether they work in academia, politics, civil society, or the private sector.

Aims and objectives

By ranking NGOs, NGO ADVISOR seeks to:

  • showcase the diversity and scale of organizations encompassed by the “NGO” label;
  • compare NGOs using criteria that transcend geography and field of activity;
  • stimulate inquiry and debate about the impact, innovation, and sustainability of NGO activity;
  • present a wide range of exemplary NGO work so that sector-wide trends can be observed year to year and annual rankings adjusted over time.

Defining “NGO”

Before we get into the specifics of our ranking methodology, we must first clarify what we mean by “NGO.” For the purpose of the Top 500 NGOs, NGO ADVISOR has revised the definition of NGOs used in the original Top 100 rankings. We have moved from the original definition…

“Operational or advocacy-focused nonprofit organizations, active at the local, national, or international level.”

… to a new definition:

“Operational or advocacy-focused entities and groups, non-profit oriented, public-interest oriented, active at the local, national, or international level.”

This change acknowledges that an entity which identifies as an NGO does not necessarily need to be incorporated, and, if incorporated, does not necessarily have to adopt nonprofit bylaws. By expanding the definition, NGO ADVISOR recognizes the eligibility of alliances, movements, public-private partnerships, coalitions, and networks without formal incorporation, as well as that of any formally incorporated entities whatever their legal form, e.g., a fund, or a for-profit that endorses a nonprofit’s objectives.

NGO ADVISOR recognizes that countless NGOs do remarkable work in areas like animal welfare, animal rights, and conservation, and the work of many ranked NGOs on human ecology has positive spillover effects for other species. NGO ADVISOR has chosen to devote its full capacity to evaluating NGOs that work on human welfare and human rights. These other fields of NGO activity are not represented in the rankings.

Key methodological developments

Though Nothias began building the ranking methodology as early as 2009, the process underwent a commissioned academic critique and review in 2013 (see “Evaluating Non-Governmental Organisations” by Cecilia Cannon, PhD, IHIED, Geneva). This review resulted in the following improvements:

  • More researchers working in different languages, thereby extending geographic reach across the NGO sector. We now operate in English, French, Spanish, Portuguese, Japanese, and Russian.
  • More NGOs surveyed for a deeper and more balanced view of the nonprofit sector, which is estimated to include close to 12 million organizations worldwide.
  • A refined definition of “NGO” with a broader application of “nonprofit,” “public interest,” and related terms to capture more activity outside the public and private sectors.
  • A process grounded in public access to NGO’s information and data. By relying on public information furnished by each NGO in its questionnaire, NGO ADVISOR demands the highest degree of transparency from the NGOs it ranks, and, in turn, places its trust in each NGO. We expect NGOs to practice accountability to the public, especially where they rely on public support.
  • A process that foregoes formal application. NGO ADVISOR asserts the value of being an independent reviewer and does not ask permission to evaluate an NGO, nor do we require NGOs to participate. NGO ADVISOR researches an NGO regardless of the NGO’s familiarity with the ranking. Nonetheless, in order to facilitate the review process, NGO ADVISOR welcomes NGOs to fill out the free online questionnaire, which is now available here. Organizations may also create a paid profile on NGO ADVISOR’s website, an option which is independent from the ranking process.
  • A codebook specifying procedures for ranking each NGO and defining detailed criteria for evaluation in order to ensure consistent, systematic, and replicable analysis.

Other additions to the rankings include:

  • A new website at ngoadvisor.net.
  • A paid subscription plan to access full website content, including the complete Top 500 ranking, instead of a paid PDF. This eliminates problems with file transfer and potential copyright issues around file duplication.
  • A set of four NGO subscription plans with benefits, including a self-managed profile in our NGO database for NGOs who opt for one of the three paid yearly NGO PRO Plans. For more information about organization subscription plans, see here.

Our evaluation criteria

NGO ADVISOR’s approach asserts that the best evaluation process only uses public data. Many NGOs publicly showcase their operations, accountability, and transparency to procure funding and support; others, however, aspire to be accountable and transparent, but may lack the means or expertise to reach their goals. Thus the questionnaire has two aims: to give NGOs a chance to demonstrate accountability and transparency and to tease out information linked to what we term our “pillars of interest.” NGO ADVISOR’s approach recognizes that excluding NGOs who do not respond to requests for information would create a distorted picture of the sector. Our research and evaluation processes are specially designed to process public information to ensure that the rankings do not misrepresent the nonprofit sector.

The three pillars of interest upon which we focus our evaluation are impact, innovation, and governance. We use the term “impact” to indicate an NGO’s output, or how it transforms the lives of its beneficiaries. In evaluating an organization’s impact, we not only look for evidence that its work has added value to the community it serves, but we also pay attention to how the organization demonstrates its efforts in its reports. We use the term “innovation” to indicate an organization’s drive to challenge itself and its ability to creatively overcome obstacles. This pillar also allows organizations with newer or more unique methods to be rewarded for their work to upset the NGO status quo. We use the term “governance” to indicate how an organization applies its good-doing mission to its employees, directors, and stakeholders.

These pillars of interest are crucial to our evaluation process. We score organizations based on their performance in 165 criteria, as explained in detail below in “Our scoring and ranking procedure.” Each criterion falls into one or more of these pillars of interest. The three main pillars of interest are weighted in the ranking process as follows: impact (450), innovation (560) and governance (390). The decision to weight impact and innovation more heavily is consistent with previous rankings, and recognizes recent arrivals to the nonprofit scene that are already having a significant impact on the landscape.

In the 2016 edition, we introduce more sub-criteria and group them into four main categories: economics and finance; marketing and communications; governance and human resources; and an overview section, which accounts for organizational history, mission, strategy, and operations, among others. We make an effort to apply each sub-criterion to each NGO according to the nature of the organization’s work. For instance, sub-criteria accounting for the expansion of an NGO’s activity cannot be applied in the same manner to two NGOs as different as, say, Partners In Health (#5 in 2013) and Wikimedia (#2 in 2013). The added complexity of these sub-criteria is counteracted by a more consistent coding rubric in the evaluation process. At this time, the sub-criteria themselves will not be published, both to protect the integrity of internal review and to ensure that NGOs do not try to gain access to these criteria and thereby gain any unfair advantage in the rankings.

An NGO and its operations are not sustainable without policies that address impact, innovation, and governance. However, transparency and accountability are not themselves pillars of interest in the evaluation. Because our scoring process assesses information that is publicly available or made available to us by an NGO, the entire exercise poses a sort of transparency and accountability challenge. It would therefore be wrong to reduce transparency to a single criterion or to introduce transparency as a pillar of interest on its own. Rather, the evaluation demands that transparency and accountability permeate an NGO’s activity. We do, however, allocate bonus points to NGOs that deserve additional recognition for their excellence in transparency and accountability.

The NGO as the unit of analysis

In evaluation and ranking, NGO ADVISOR uses the NGO as the unit of analysis, rather than program- or intervention-level parameters common in development studies or within donor and practitioner communities. While uncommon in most of the development sector, the assessment of entire organizations is well established among political scientists. This choice also reflects both the breadth of cross-sector, international NGO rankings and responds to scholars’ call for wide-angle views of organizations.

Researching NGOs

NGO ADVISOR selects NGOs through various media, social media, databases, and introductions via email or other means. We carefully sift through databases like ECOSOC, as many listings are inactive, pro-business, pro-government, or simply irrelevant to nonprofit research. Despite the biases of agenda-driven groupings, databases like NGO Monitor can be worthwhile sources. NGO ADVISOR also initiates research through straightforward contact; many NGOs reach out to introduce themselves. To date, we have considered a total of roughly 3,800 NGO listings in our search for organizations.

Internet presence is an important factor in our selection of NGOs. Researchers often establish contact either via an organization’s website, email, social media, or other online forms. Some NGOs provide email contacts for press inquiries; others may have a basic website contact form connected to an organizational email account. Attempts to contact NGOs are occasionally blocked by spam filters. Some attempts go entirely unanswered. Even when an organization can be reached, establishing correspondence with the “right” staff member can be challenging. Due to the cost of high-volume international calling, we avoid phone contact. Overall, the challenge of getting in touch is a concern, but not a great one; research processes have been refined to utilize information that can be gathered through various channels and not by personal contact alone.

The 2015 round of rankings benefited from Global_Geneva’s establishment of a global network of researchers working across a wide range of languages. This network plays a critical role in the systematic survey of NGOs, as we are now unhindered by language filtering and a priori cultural reasoning. Previous rankings suffered from a predominance of English-speaking researchers. This network has already yielded encouraging results in eliminating bias towards organizations from the Global North in the evaluation process.

Nominating NGOs

The research phase identifies NGOs to enter the review process. NGO ADVISOR encourages organizations to enter the evaluation process by filling out the online questionnaire. However, a complete questionnaire is not mandatory for evaluation; we review organizations whether or not they elect to be evaluated. In cases where NGO ADVISOR relies solely on public records to complete its review, the quality of this information is taken into consideration in NGO ADVISOR’s evaluation of an NGO’s transparency and accountability. In the absence of public information, NGO ADVISOR adapts its review to consider whether there may be good reasons for a lack of transparency.

For the 2015 round of rankings, we considered imposing a registration fee as a way of building support for the rankings. Because a fee could limit NGO participation, however, we ultimately rejected this idea. We also rejected the option to raise funds from philanthropic foundations, as many of them also support organizations in the ranking. By offering online services to NGOs and its readership, NGO ADVISOR has created a new way to preserve its independence. Some NGOs recommended this approach in 2014 and we thank them for putting us on the right track.

Evaluating NGOs

Because our scoring process does not focus on detailed case studies or on organizations’ internal evaluations of their progress, we can instead focus on the fair and independent review of information gathered through 1) entity website(s) and 2) NGO ADVISOR’s questionnaire.

The public information provided by each NGO is very useful. These data provide a view of an NGO’s transparency and accountability that is potentially more informative than private statements made in a confidential questionnaire. At the same time, we recognize that some NGOs may be working toward greater accountability and transparency, or may have reasons for non-disclosure. We have opted to make all NGOs’ questionnaire responses available to our subscribers and visitors. NGOs may control the full extent of their transparency by opting for a Registered, Verified or Certified NGO PRO status. For more information about PRO status, please see here.

We anticipate that the information in questionnaire responses will often, but not always, be provided along with links to its sources. NGO ADVISOR must deal with the varying degrees to which information about NGOs is verifiable. One of our goals is therefore to spur dialogue about the degree to which NGOs, insofar as they seek public support, should be held to standards of public accountability.

Our scoring and ranking procedure

The scoring procedure uses the following numerical scales:

  • a score of 0 or 5, where 0 indicates “NO” or an absence of information and 5 indicates “YES”
  • a scale of 0 to 5, with 0 corresponding to low performance and 5 to high
  • a simple count (e.g., a point for each year an organization has been active, capped at 12 points)
  • a few critical criteria are multiplied by 4

The maximum score is 1400 points. The number of criterion has grown from 145 in the 2015 ranking to 165 in 2016. An organization’s total score can be increased or reduced by bonus points/penalties:

  • as many as 100 bonus points for independence, transparency, accountability, and for the quality of information provided in the questionnaire (e.g., do questionnaire answers include links to public data sources)
  • as much as an 80 point penalty for dependence on corporations, governments, single funders, or other specified sources.

The simplified numerical scales have minimized the risk of subjective inconsistencies in scoring, which the 2013 discussion paper highlighted as an area of improvement. In 2015, we introduced a scoring system for previously ranked NGOs (1 to 100) to ease the transition from a Top 100 to a Top 500 and to avoid inconsistencies between the 2013 and 2015 editions. We will not apply this transition scoring in the rankings published in 2016 or later.

In order to obtain a final ranking, we create an overall index score that has a scoring range sufficient for allocating different ranks to 500 entities. Each organization receives an aggregated overall score on a numerical scale from 1 to 1400 to minimize the risk of attributing the same rank to any two NGOs.

Researchers and reviewers

The confidentiality of researchers and reviewers is critical to any review work. With the exception of our registration desk, which maintains open and direct communication with participants, NGO ADVISOR observes a strict confidentiality policy.

In the process of improving our methodology, we have also simplified the roles of researcher and reviewer and minimized the risk of subjective inconsistencies to the greatest extent possible. Previous rankings were conducted by Masters students from the Geneva area, specifically IHEID and UNIGE. Thanks to the improved methodology, our review staff now includes Masters and PhD students from a wide variety of locations and languages.