Methodology

Introduction

The concept of “non-profit” has evolved far beyond traditional charity work to encompass myriad associations, foundations, coalitions, alliances, movements, and even businesses that serve a diversity of objectives and engage in many ways with public and private institutions. NGO ADVISOR monitors the international non-profit community for ideas, values, and models that challenge normal approaches to policy, the market, and NGO activity. A key monitoring tool is the Top 500 NGOs ranking. This ranking grows out of The Global Journal Top 100 NGOs rankings, which have been expanded to the Top 500 and moved to NGO ADVISOR.

NGO ADVISOR is currently chaired by Jean-Christophe Nothias, former editor of The Global Journal, who brings to the table years of journalism experience in politics, policy, and social development. Through a journalistic approach to research and reportage, NGO ADVISOR uses the rankings as a lens to bring into focus the social transformations of the NGO sector, and to magnify the broad and evolving range of values that non-profit work generates, values all but invisible from the standpoint of stock indices and other revenue-based perspectives. NGO ADVISOR’s overall goal is to engage and encourage thought leaders and remarkable innovators interested in the power of the non-profit concept, whether they come from academia, politics, civil society, or the private sector.

 

Aims and Objectives

By ranking NGOs, NGO ADVISOR seeks to:

  • showcase the diversity and scale of organizations encompassed by the ‘NGO’ label;
  • compare NGOs using criteria that transcend geography and field of activity;
  • stimulate inquiry and debate about impact, innovation, and sustainability of NGO activity;
  • present a wide range of exemplary NGO work so that sector-wide trends can be observed year to year and annual rankings adjusted over time.

 

Key Developments and Improvements

In 2013, the rankings benefited from a commissioned academic critique and review (see “Evaluating Non-Governmental Organisations” by Cecilia Cannon, PhD, IHIED, Geneva), which resulted in the following improvements:

  • More researchers in different languages extending geographic reach across the NGO sector (English, French, Spanish, Portuguese, Japanese, and Russian).
  • More NGOs surveyed (over 2,000) for a much deeper and more balanced view of the non-profit sector, for which the number of entities varies from 3,500 to 65,000, depending on the source.
  • A refined definition of “NGO” with a broader application of “non-profit”, “public interest”, and related groupings to catch more activity outside of private and governmental sectors.
  • A process grounded in public access to NGO’s information and data. Where journalism relies on fact-checking, the NGO rankings rely on statements and data furnished by the NGOs themselves, which admit of fact-checking to varying degrees. By relying on public information, NGO ADVISOR expects the highest degree of transparency and accountability from the NGOs it ranks, and, in turn, places its trust in each NGO. A corporate IPO requires a high degree of public accountability from the corporation; similarly, NGO ADVISOR encourages the public to have a stake in NGOs, and expects NGOs to practice accountability to the public, especially where they rely on public support.
  • A process that foregoes formal application. NGO ADVISOR recognizes the value of being an independent ranking editor, and therefore does not ask permission to evaluate an NGO, nor are NGOs asked to apply or participate. NGO ADVISOR researches an NGO regardless of the NGO’S familiarity with the ranking. Still, in order to facilitate its works and supports its efforts, NGO ADVISOR welcomes and invites NGOs to fill the online questionnaire now at their disposal, but this is not to be understood as an application. Whether an NGO feels appropriate to create a profile in NGO ADVISOR’s website is independent from the ranking process however, all accurate and up-to-date information are welcome.
  • A codebook detailing procedures for ranking each NGO. Among the changes in methodology for evaluating NGOs, one suggestion in the discussion paper was to use “a documented codebook clearly articulating the criteria for evaluation that facilitates systematic and consistent analysis by all staff members and across the evaluation period. Documenting detailed information on what each criteria and sub-criteria entail, including details on what a low score for each criterion looks like, and what a high score for each criterion looks like, not only facilitates consistent and more rapid ‘coding’ of NGOs in the evaluation process, but would also permit an external actor to assess any of the NGOs in the ranking using the same criteria, and potentially replicate the results.”

From now on, the scoring procedure includes the following ordinal scales:

– 0 or 5 (for ‘YES’ or a ‘NO’)
– 1 to 5 (regular scale)
– 0 to 5 (‘0’ for ‘no’ or an absence of score)
– x 4 when criterion is critical
– basic count (e.g., for a number of years, usually with an upper limit such as “max 10” or “max 20 years”)

The maximum score is now 1400 points. Number of criterion for 2016 is 165 when it was 145 for the 2015 edition. Total of points can be increased or reduced by bonus/malus points:

– as many as +100 bonus points for independence, transparency, accountability, and for the quality of the questionnaire (e.g., do questionnaire answers include links to public data sources)

– as many as -80 malus points for dependence on corporations, governments, single funders, or other specified sources.

Other additions to the rankings include:

  • A new website: www.ngoadvisor.net
  • Subscription to access all contents, instead of a paid-for PDF, precluding troublesome file transfer and any copyright issues around file duplication.
  • Set of subscription for NGOs that includes:

> A self-managed profile in our NGO database for NGOs who opt for one of the three yearly NGO PRO Plan (paid for). Please read more of the different subscription plans in the ‘Add Org’ section of the website.

 

Researching, nominating, and evaluating NGOs

NGO ADVISOR’s approach assumes that evaluation can only use accessible or public data. To procure funding and support, most NGOs showcase their operations, accountability, and transparency. Sometimes, there is desire for accountability and transparency, but not the means or expertise. Thus the questionnaire has two aims: to tease out information linked to the three criterial areas—impact, innovation, and governance—and to give NGOs a chance to demonstrate accountability and transparency. In a sense, “governance” entails transparency and accountability; an NGO is not sustainable without policy that addresses these issues. However, transparency and accountability are not themselves criteria in the evaluation. They do factor into the scoring process as “bonus” scales in which an NGO may garner additional recognition for excellence in these areas, but rather than establish them as key criteria in the ranking, NGO ADVISOR regards transparency and accountability as fundamental obligations for any serious NGO.

NGO ADVISOR’s approach also recognizes that excluding NGOs who do not respond to requests for information would create a distorted picture of the sector. Research and evaluation processes specially designed to process public information ensure that the rankings do not misrepresent the sector in this way.

 

Definition of “NGO”

For the purpose of the Top 500, NGO ADVISOR has changed the definition for NGOs.

From the original definition:

Operational or advocacy-focused non-profit organizations, active at the local, national, or international level.

To a new definition:

Operational or advocacy-focused entities and groups, non-profit oriented, public-interest oriented, active at the local, national, or international level.

This change acknowledges that the entity does not necessarily need to be incorporated, and, if incorporated, does not necessarily have to adopt non-profit bylaws. By enlarging the definition, NGO ADVISOR recognizes the eligibility of alliances, movements, public private partnerships, coalitions, and networks without formal incorporation, as well as that of any formally incorporated entities whatever their legal form, e.g., a fund, or a for-profit that endorses a non-profit’s objectives.

NGO ADVISOR recognizes that countless NGOs do remarkable work in areas like animal welfare, animal rights, and conservation, and the work of many ranked NGOs on human ecology has positive spillover effects for other species. Because NGO ADVISOR must devote its full capacity to evaluating NGOs that work on human welfare and human rights, these other fields of NGO activity are not represented in the rankings.

 

The NGO as the unit of analysis

In evaluation and ranking, NGO ADVISOR uses the NGO as the unit of analysis, rather than program- or intervention-level parameters common in the developmental studies literature or within donor and practitioner communities. This choice reflects both the breadth of the cross-sector, international NGO rankings, and a response to scholars’ call for wide-angle views of organizations. While uncommon in most of the development sector, organization-wide assessment is well-established among political scientists; in fact, one assessment for lobby/NGO advocacy comes directly from policy practices, and has gained traction in national, and now increasingly, international ranking and evaluation.

 

Researching NGOs

NGO ADVISOR selects NGOs through various media and social media, databases, and listings, and through introductions via email and other means. A database like ECOSOC requires sifting; many of its listings are either inactive, pro-business, pro-government, or simply irrelevant to non-profit research. Despite the biases of agenda-driven groupings, databases like NGO Monitor can be worthwhile sources. NGO ADVISOR also initiates research through straightforward contact; many NGOs simply reach out to introduce themselves.

Websites are an important factor in our selection of NGOs. In working through over 3’800 NGO listings, researchers had to establish contact either via email (whether through a website or not), social media (e.g., through a Twitter account), or via some other online form that provided access to the NGO. Some but not all NGOs provide email contacts for press inquiries. Some may have a basic contact form connected to an organizational email account. Attempts to contact NGOs are occasionally blocked by spam filters. Some attempts go entirely unanswered. Even when an organization can be reached, establishing correspondence with the “right” staff member can be challenging. Due to the cost of high-volume international calling, phone contact is avoided. Overall, the challenge of “getting in touch” is a concern, but not a great one; research processes have been refined to utilize information that can be gathered through various channels, not by personal contact alone.

The third round of rankings benefits from the establishment of a global network of researchers working across a wider range of languages, a change critical to a systematic survey of NGOs unhindered by language filtering and a priori reasoning. Previous rankings suffered from a preponderance of English-speaking researchers. As mentioned in the discussion paper, “While the Top 100 NGOs list does include NGOs beyond the Geneva/ New York/Washington DC hub, there remain gaps of representation.” Enhanced localization has already yielded encouraging results.

 

Nominating NGOs

The researching phase identifies NGOs to be nominated and enter the review process. NGO ADVISOR facilitates review with its online questionnaire. However, a completed questionnaire is not mandatory for review and evaluation. Where NGO ADVISOR has to rely on public information, the quality of this information is taken up in NGO ADVISOR’s evaluation of an NGO’s transparency and accountability. In the absence of public information, NGO ADVISOR adapts its review to consider whether, as in the case of Wikileaks, there may be good reasons for a lack of transparency.

For the 2015 round of rankings, a registration fee was considered as a way of building support for the rankings, but because a fee could limit NGO participation, it was ultimately rejected. By offering services to NGOs and its diverse audience, NGO ADVISOR has set a new way to preserve and support its independence in establishing the ranking. Search for funds from foundations and philanthropes was not retained as a good option, as many of them do support specific organizations part of the competition. Few NGOs have recommended at an early stage back in 2014 to opt for this approach and we thank them for putting us on the right path on this.

 

Evaluating NGOs

Because the evaluation does not focus on detailed case studies, nor on process-tracking and attainment of goals assessed by other methods, it can focus on fair and independent review of information gathered through 1) entity website(s) and 2) NGO ADVISOR’s questionnaire.

The public information provided by each NGO is useful in many regards: it provides a look at an NGO’s transparency and accountability, one that is potentially more informative than private statements made in a “confidential” questionnaire (while recognizing that, as above, some NGOs may be working toward greater accountability and transparency, or may have reasons for non-disclosure). This raises the question of whether a policy or formal statement should address the possibility of questionnaire responses being published by NGO ADVISOR. We have opted for making the questionnaire responses available to our readers and visitors, with possibility for NGOs to decide of the extent of their own transparency paving the way to new status in our platform. NGOs might opt for a ‘registered’, ‘verified’ or ‘certified’ status in terms of public profile, meaning that part of the questionnaire responses might differ from one status to another. So we decided to have registered organizations in our platform responsible for their level of transparency and accountability. For more on this, please read the ‘Add Org’ section online.

Information in questionnaire responses is often, but not always, provided along with links to its sources. NGO ADVISOR must deal with the varying degrees to which information and data about NGOs is verifiable. Thus NGO ADVISOR’s interest in spurring dialogue about the degree to which NGOs, insofar as they seek public support, should be held to standards of public accountability.

 

Scoring and ranking NGOs

The evaluation now uses the following ordinal scales for sub-criteria:

– 0 to 5 for most sub-criteria
– 0 or 5 (when related to a ‘YES’ or a ‘NO’)

The simplified ordinal scales (0 to 5) and (0 or 5) have minimized the risk of subjective inconsistencies in scoring, which the discussion paper highlighted as an area of improvement. There are no longer scores from 1 to 20 on any sub-criteria.

In 2015, a scoring for previously ranked NGOs (1 to 100) has been introduced to aid transition from a Top 100 to a Top 500 system and to avoid inconsistencies between the 2013 and 2015 editions. This ‘transition’ scoring is no longer in application for the 2016 ranking.

More sub-criteria have been introduced. The added complexity is far outweighed by the more consistent and rapid “coding” of NGOs in the evaluation process. Sub-criteria are grouped into four main categories: economics and finance; marketing and communications; governance and human resources; and an overview section, which accounts for organizational history, mission, strategy, and operations, among others. The sub-criteria within these four main categories are organized into the three criterial areas of impact, innovation, and sustainability in order to represent NGOs in these terms. An effort is made to apply each sub-criterion to each NGO according to the nature of the NGO. For example, sub-criteria accounting for expansion of an NGO’s activity cannot be applied in the same manner to two NGOs as different as, say, Partners In Health (#5 in 2013) and Wikimedia (#2 in 2013).

At this time, the sub-criteria themselves will not be published, both to protect the integrity of internal review and to ensure that NGOs do not gain access to these criteria and thereby gain any unfair advantage in the rankings.

 

Researchers and reviewers

The confidentiality of researchers and reviewers is critical to their work. One observes this in rankings across a number of fields; from gastronomic guidebooks to politics to business (e.g., The Economist has a policy about bylines that arises from similar concerns). NGO ADVISOR observes this policy of confidentiality with the exception of the registration desk, which maintains open and direct communication with participants.

The roles of researcher and reviewer have been simplified by improvements in methodology, and the risk of subjective inconsistencies has been minimized to the greatest extent possible. The previous rankings were conducted by Masters students from the Geneva area (IHEID and UNIGE). Thanks to improved methodology, rankings now involve Masters and PhD students from a wide variety of locations and languages.

 

Final ranking

In order to obtain a final ranking, it was necessary to have an overall index score that had a scoring range sufficient for allocating different ranks to 500 entities. Therefore, each entity received an aggregated overall score in an ordinal scaling from 1 to 1400 to minimize risk of attributing the same rank to any two NGOs. The three main criteria are weighted as follows: impact (450), innovation (560) and governance (290). Bonus/malus points can increase a score by +100 bonus points for independence, transparency, accountability, and for the quality of the returned questionnaire, and can decrease a score by -80 points for dependence on corporations, governments, single funders, or other specified sources.

The decision to weight impact and innovation more heavily is consistent with the previous rankings, and recognizes entities that recently entered the non-profit sector but are already having a significant impact on the landscape.

 

Advisory board

To anticipate any issues and questions raised by NGOs in the ranking process, and to engage the broadest constituency possible in creating fair, independent, and high-quality rankings, Global_Geneva, the non profit that published the 2015 edition, will maintain an advisory board to address the ideas, input, and questions of participants, and work toward the constant improvement of the Top 500 NGO rankings.