Update (11th April 2020): This paper has been accepted at ACM Web Science 2020. A preprint can be accessed on arXiv.
Nation states around the world engage in web censorship using a variety of legal and technical methods. India is no different in this regard: the Government of India can legally order internet service providers (ISPs) operating in its jurisdiction to block access to certain websites for its users. This makes the situation different from jurisdictions like Iran and China, where internet censorship is largely centralised. Legal provisions in India, namely Section 69A and Section 79 of the Information Technology (IT) Act, allow the Central Government and the various courts in the country to issue website-blocking orders that ISPs are legally bound to comply with. Most of these orders are not publically available.
Recent events and the opaque nature of internet censorship in India motivated us at The Center for Internet and Society to study India's censorship mechanism in detail. We spent the last year trying to answer two questions pertaining to how internet users in India experience web censorship:
- What are the technical methods of censorship used by ISPs in India?
- Are all ISPs blocking the same websites?
Our work has been so far the largest study of web censorship in India, both in terms of the number of censorship mechanisms that we test for and the number of potentially-blocked websites (PBWs).
Data curation
We compiled a list of PBWs from three sources:
- Government orders: A website/URL blocking order may come from the Government of India (Section 69A, IT Act). These orders are usually not in the public domain, as a confidentiality clause prevents any party from disclosing its contents. We collect published and leaked Government orders.
- Court orders: The various courts in India also have the power to issue website blocking orders (Section 79, IT Act). Not all such orders are available in the public domain. However, the Government and BSNL (a public company operating as an ISP) have provided portions of this list when under pressure to respond to Right to Information (RTI) requests.
- User reports: The Internet Freedom Foundation collects and publishes reports from internet users who notice blocked websites.
Collecting data from these sources led to a total of 9673 unique URLs, which yielded 5798 unique websites. To limit ourselves to active websites, we exclude all websites for which we could not resolve via Tor circuits, culminating in a corpus of 4379 PBWs.
Network tests for detecting censorship
We designed four network tests that probe the existence of censorship at the DNS, TCP, HTTP, and TLS level. For the sake of brevity, I'll skip elaborating on the tests in this post; the details can be found in our preprint.
We run these tests for each website in our corpus from connections of six different ISPs (Jio, Airtel, Vodafone, MTNL, BSNL, and ACT), which together serve more than 98% of Internet users in India. Our findings not only confirm that ISPs are using different techniques to block websites, but also demonstrate that different ISPs are not blocking the same websites.
Results
In terms of censorship methods, our results confirm that ISPs in India are at liberty to use any technical filtering mechanism they wish: there was, in fact, no single mechanism common across ISPs.
We observe ISPs to be using a melange of techniques for blocking access, such as DNS poisoning and HTTP host header inspection. Our tests also discern the use of SNI inspection being employed by the largest ISP in India (Jio) to block HTTPS communication, the use of which is previously undocumented in the Indian context.
Further, we notice that all ISPs using multiple censorship mechanisms are not blocking the same websites with each mechanism. For instance, ACT uses only DNS censorship for blocking 233 websites, only HTTP censorship for 1873 websites, and both to block 1615 websites. Such irregularities are illustrated below.
Some alarming discoveries
Our study has recorded large inconsistencies in website blocklists of different Indian ISPs. From our list of 4379 PBWs, we find that 4033 are being blocked by at least one ISP’s blocklist. In terms of absolute numbers, we notice that ACT blocks the maximum number of websites (3721). Compared to ACT, Airtel blocks roughly half the number of websites (1892).
Perhaps most surprisingly, we find that only 1115 websites out of the 4033 (just 27.64%) are blocked by all six ISPs. Simply stated, we find conclusive proof that Internet users in India can have wildly different experiences of web censorship.
Analysing inconsistencies in blocklists also makes it clear that ISPs in India are:
- Not properly complying with website blocking (or subsequent unblocking orders), and/or
- Arbitrarily blocking websites without the backing of a legal order.
This has important legal ramifications: India’s Net Neutrality regulations, codified in the license agreements that ISPs enter with the Government of India, explicitly prohibit such behaviour.
Our study also points to how the choice of technical methods used by ISPs to censor websites can decrease transparency about state-ordered censorship in India. While some ISPs were serving censorship notices, other ISPs made no such effort. For instance, Airtel responded to DNS queries for websites it wishes to block with NXDOMAIN. Jio used SNI-inspection to block websites, a choice which makes it technically impossible for them to serve censorship notices. Thus, the selection of certain technical methods by ISPs exacerbates the concerns created by the opaque legal process that allows the Government to censor websites.
Summing up
Web censorship is a curtailment of the right to freedom of expression guaranteed to all Indians. There is an urgent need to reevaluate the legal and technical mechanisms of web censorship in India to make sure the curtailment is transparent, and the actors accountable.
The whimsical attitude towards web censorship from both ISPs and the Government necessitates the development of a crowdsourced tool to monitor and measure such censorship from different vantage points in the country. This will shed further light into the geographical variation of censorship practices by ISPs across India, which is still unclear.
To probe this further we have ported our network tests into an android application, and are looking for volunteers who are willing to run it on their mobile networks. The entire process will be completely anonymous; we will not be collecting any user-specific information. If you live in India, please consider running Censorwatch.
Acks - This study was done in collaboration with Gurshabad Grover and Varun Bansal at The Center for Internet and Society, graciously supported by the MacArthur Foundation.