When I started Mamushi Mobile, the aim was to assist individuals and companies with being able to privately acquire secure devices that were free of censorship. It was clear there was market demand for this and, having used a variety of operating systems and devices over the years, the latest Pixel phones installed with “de-Googled” software were a game-changer. They had a secure chip together with an ability to support custom firmware signing keys, were excellent value for money and had regular security updates.
Last month I terminated the reseller agreement with Copperhead and will instead be pivoting the business in the interim to selling devices with the popular open source CalyxOS software pre-installed. Although Mamushi is not associated with The Calyx Institute, our small team is confident of our ability to support customers with the software and fulfill the needs of our small but growing market that want a hassle free, frictionless way of obtaining and using a secure and private phone. In the future, we may look to host our own Android Open Source Project (AOSP) distribution but, for the moment, after a few weeks of active use and testing, we are very impressed with CalyxOS. This was not a decision made in haste and, although I am disappointed with Copperhead, I am very much encouraged by the support I have had so far from those who I have informed already of this decision. It also gives us the chance to promote and use software closer to our ideological roots. Many expressed reservations about Copperhead when Mamushi was first launched but gave me the benefit of the doubt.
Prior to setting the company up I, along with many other in the Samourai Wallet community of beta testers, contributors, privacy advocates and users were already exploring alternative ROMs that offered better privacy and security features than stock Android. CopperheadOS had been the choice in 2017 of a handful of users I knew but, with the co-founder leaving abruptly in 2018 (founding GrapheneOS), it had left quite a void. Devices were left without security updates for an extended period of time and serious allegations were hurled between co-founders alleging fraudulent handing of funds, destruction of signing keys, deletion of code repositories, and providing access to third parties. They even dispute their respective roles and, as of today, there is still ongoing litigation.
I, personally, had little interest in the drama between the co-founders as I had not been a customer but I was keen on using an Android distribution that could offer close to the security and privacy of iOS when I began testing a privacy focused Bitcoin wallet. CopperheadOS was only supported on the Pixel 2 range of devices in December 2019 which were clearly inferior to the Pixel 3 and 3a range, so I installed GrapheneOS and began using that in earnest. Overall, I found the software to be relatively stable but somewhat slow, there were some understandable UX trade-offs yet it was clearly very secure and private. Many Bitcoin users, including myself, started recommending the OS and the applications that could be run on it.
Although the OS had compromises that made it difficult to be used as a replacement for an ordinary phone, many of the community adopted it. It grew further especially when popular video guides helped users overcome the challenge of some barebones instructions on the website. I had hoped to see further development of the OS to support a full encrypted back-up and restore (especially useful for journalists crossing borders), an ability to run trusted applications without being killed off in the background and to circumvent some of the restrictions that Google had been introducing that had a direct impact on Bitcoin users.
Unfortunately, it became quite clear to me that the lead maintainer of GrapheneOS was not stable. I was happy to donate to the project and was interested in sponsoring some open source development. Despite the fact that he was at the forefront of Android security, he was using an insecure device as well as a static donation address for receiving Bitcoin. My efforts were rebuffed, I was accused of being complicit in harm and abuse for merely mentioning his co-founder and also told I wasn’t welcome to use the OS that was under a range of open source licenses such as GPLv2, Apache and MIT as well as, curiously, some closed source licensing. Months later CopperheadOS was described as “only AOSP with custom boot animation… tracking… a fashion accessory”, “less private and secure than AOSP” yet prior to this it was described as including the hardened Android work and violating a non-commercial clause which was clearly contradictory.
Moving to CopperheadOS from GrapheneOS
The behaviour on social media from the lead maintainer of GrapheneOS was, to my surprise, enough to turn many other users off running the OS and I received a number of concerned messages. This was, coincidentally, shortly after CopperheadOS was supported on the Pixel 3 and 3a so, with Copperhead looking for some testers and feedback on the OS, I set up a beta testing group and a handful of users were provided access to the ROM to install . No phones were provided to anyone by Copperhead. We simply installed the OS on our own devices and provided feedback. Installation was easier than GrapheneOS, the device appeared to perform a tad faster and F-Droid was included in the OS which made for less friction on initiation.
Overall, there was little apparent difference for the user between CopperheadOS and GrapheneOS. The obvious omission of Google Mobile Services meant certain apps had reduced functionality. One notable exception, however, was the lack of the Auditor app despite being documented on Copperhead’s site. I raised this immediately and was assured that this was “on the roadmap”. It has never been implemented. Although I was somewhat concerned with this at the time, I had also pointed out that the site was incorrectly stating CopperheadOS was open source and this was corrected soon after to be “source available”. I believed my comments, as well as the other beta testers, were being taken seriously; especially with regards to the then newly integrated app, SeedVault (built by CalyxOS developers) which was a key requirement of ours. Mamushi Mobile was launched soon after.
Asia was a largely untapped market for Copperhead resellers and I was hopeful that revenue for Copperhead could result in more resources being dedicated for development. I also enjoyed the challenge of selling a product in the wake of Copperhead’s abysmal social media presence. Unfortunately, the Copperhead sales partner network was disappointing. Despite asking for linkbacks to their site, Copperhead were reluctant to link their site to Mamushi Mobile or other resellers. Prospective customers were repeatedly confused when arriving at the Copperhead site as it was not clear to them how to purchase a device, whether to download the software to install themselves or even know which companies were authorised resellers when they were contacted by us when sent over as “leads”.
For the retail business, software licenses were, bizarrely, required to be purchased in bulk as though they were physical goods and, once a package was sold, minimum seller retail prices (MSRP) were then dropped so the distributor of the licenses had an advantage over the resellers. There were no geographical boundaries between resellers and I even discovered active undercutting of MSRPs despite having complained once already and doing business under a sales agreement that ostensibly prevented “lowballing”. Mamushi’s distributor even offered direct download of the software using a private flasher which resellers were prohibited from doing. The sales strategy then took a turn for the worse with inept and dishonest marketing around CopperheadOS vs GrapheneOS attributes. Feature comparisons had long been requested from prospective customers, so I was angered enormously to see this hack job. A whole article could be written on how misleading and incorrect the page was and remains.
Moreover, product development was glacial. Although Android 11 brought noticeable performance improvements, there was nothing to suggest that this was the work of Copperhead developers themselves. During beta testing, it was brought to Copperhead’s attention that Google suggestions remained in the browser, an embarrassing oversight in a product of which the core value proposition was avoiding sending data back to Google. The software was released to production anyway with the bug still present and not fixed until a month later. SeedVault no longer worked on this update and subsequent updates failed to fix the application. For a paid OS that was supposedly providing regular updates, September and October security patches were delayed until November. Support for the Pixel 4 and 4 XL was announced on the 3rd December (without notice to resellers to prepare inventory) four months after the devices were discontinued and five months after GrapheneOS had released support.
The final straw however, was on the corporate side. The behaviour from the CEO made it quite clear that they were not a partner that could be relied upon. When I brought in new business in the form of a six figure prospect that was an ideal fit for the company, arranged calls and personally authored a sales proposal, the contribution from Copperhead (after quite some delay) was to add redundant boilerplate and, in an egregious move, attempt to remove Mamushi as a legal entity from the deal. It was argued that I didn’t have the license to broker the deal I had been requested to bring yet no new contract was provided despite my willingness to agree to one. Copperhead also raised concerns about my proposed pricing as being too high which turned out to be entirely unfounded when, after stonewalling with negotiations, the revised pricing for the prospect was increased even further and the commission they were willing to pay to Mamushi was reduced by 90%. Copperhead were only prepared to do the absolute minimum in development work for the client, charge outrageous amounts for support whilst expecting a sales funnel and ongoing relationship management from Mamushi at the commission rate of a payment processor. In my 15 years of working in software delivery, I had never seen such amateurism, underhandedness, and poor value for money.
With the litigation between Copperhead shareholders progressing, a customer of mine who had, of his own volition, decided to do a write-up of his experience purchasing a device, was targeted as part of an ongoing harassment campaign from GrapheneOS. Although not susceptible to being bullied off his operating system of choice, he was uncomfortable with some of Copperhead’s prior behaviour. With CalyxOS being well reviewed and gaining in popularity, he flashed his CopperheadOS device and then detailed an excellent summary of his experience. Shortly afterwards, I did the same and the beta testing group I had set up a year prior to try out CopperheadOS moved en masse without any encouragement from me.
How does it all compare? In truth, the objectives of the CalyxOS project are not the same as the GrapheneOS project which offers, unsurprisingly, a similar user experience to CopperheadOS given its origins. I have, however, concluded though that the CalyxOS device is far more usable and functional than the other operating systems whilst achieving what hitherto hasn’t been possible: a consumer facing mobile device that is actually secure and privacy respecting. This isn’t to denigrate the security research being done by the teams which provide users with genuine security improvements to Android in the areas of memory allocation, runtime and process spawning but these come at quite a performance cost.
The developers will allege that these compromises are not noticeable on more modern hardware but our experience has proven otherwise. Battery performance is noticeably worse and switching between apps for example, say from a browser to a mobile wallet with a messaging client open can often end up with apps in the background being killed. If you are you using these for circumstances like cryptocurrency transactions, these issues can be frustrating, time consuming and, in the worst case, costly. CalyxOS isn’t missing Android hardening entirely as it benefits from the standard protections in AOSP which has made use of “upstreamed” security enhancements over the years but this has been done without compromising as much on performance and optimisation of memory usage.
Furthermore, other design choices in GrapheneOS and CopperheadOS now seem somewhat dated. The Secure PDF reader may well offer protection when opening PDFs but no one is likely to use it. Zooming of pages for reading is poor and page viewing is awkward with no easy scrolling. The benefits of hardened Chromium and Vanadium will go unused by most users for browsing the modern web as they have no inbuilt adblocker and, although one could use NetGuard for blocking adverts at a network level, you would then be left without the ability to use a commercial VPN to obfuscate your IP and hide your browsing activity from your ISP. Users will just typically install the Bromite browser.
CalyxOS, on the other hand, includes useful privacy features “out of the box” that you would expect from an operating system built around a customer. Whereas the home screens of GrapheneOS and CopperheadOS are rather bare (with a glaring lack of UI polish) CalyxOS assists the user with installation of a whole host of useful apps including free commercial VPNs, a distress button for wiping apps and includes Signal so users have this very popular secure messenger without needing to download an APK or go via the Aurora store. This ensures users can benefit from the mitigations F-Droid provides which is an essential part of the security of the device. The ability to use microG allows greater Android app compatibility, eSIM support and notifications for core apps such as Protonmail, RocketChat and Keybase as it is compatible with Google Cloud Messaging which can be a dealbreaker for many.
Whether these choices are appropriate for an individual’s own threat model is up to them to decide. Some will always go for what is perceived as the securest device even if it is not practical for use on a day-to-day basis. My suggestion, however, would be that if you are a target of a “three letter agency”, you would be best to avoid carrying a mobile device altogether or do so with most of the telemetry removed anyway. At Mamushi, we are “mobile first”. If a user can’t perform their duties without carrying a second phone or being tethered to a computer, we see this as a cop out. The securest device for a user is the one they will actually use.
For high threat environments, a Pixel with CalyxOS powered off offers remarkable security against brute force attacks. In use with an end-to-end encrypted messenger, users should have a high degree of confidence that their communications are protected against surveillance and interception. Of course, individuals must always be prudent with the software they install, and the paranoid can opt to open sensitive apps on Android secondary profiles (especially on first boot) to mitigate extreme concerns over possible exploits.
I am very keen to put my experience with the Copperhead shareholders firmly in the rear-view mirror. The litigation has been enormously damaging for them and I am not optimistic that a court ruling will bring any finality to the situation. Prior to termination of the reseller agreement, I had lobbied hard for CopperheadOS to move to an open source license such as AGPL. They have remained resolute in their resistance to this while on a quixotic mission to stop the spread of “pirated software”. Code needs to be free. Aside from the impracticality of this and their complete failure so far to stop the widespread distribution of what they perceive as their intellectual property, Copperhead’s business is one of charging for support and assisted deployments which aligns well with open source software. Although I have seen nothing to suggest backdoors are present, allegations are hard to shake off when they are not forthcoming with the availability of the source code.
At Mamushi, we are looking forward to moving onto greener pastures. From now on, we will only be partnering with open source projects and businesses. We are keen to sponsor open source developers for specific Android work so don’t hesitate to get in touch if you think you have something to offer. Our immediate priorities will be to improve our support and documentation as well as to host some services ourselves for customers.