Stop allowing people to use DO droplets for mining digital currencies
CPU Steal time has been high for the last 4 droplets i created (average 15% with spikes up to 50% cpu steal), these droplets are really slow compared to my other droplets with 0-5% cpu steal time.
I'm seeing a lot of people coming over to DO the last few months creating droplets, sometimes 50+ droplets per person, using them to mine digital currencies and use 75-100% cpu 24/7
We’ve updated the backend to provide more fair sharing of CPU specifically to account for cases of mining.
Luckily the high value of Bitcoin has drastically dropped in the last 6 months and we’ve seen this reflected in a much smaller number of active users mining bitcoins.
Much of the mining was happening by fraudulent accounts as the economics of mining bitcoins or other cryptocurrencies weren’t financially feasible when there are server farms based on GPU systems that were driving the valuation along with the overall speculation in the marketplace.
Hopefully customers have seen a reduction in the amount of steal time as a result and if you are still seeing a high percentage of that please open a ticket and we’ll review the hypervisor in question to see if there are any abusive accounts that need to be addressed.
Thanks very much for the feedback!
It seems that this problem hasn't been solved yet. One of our servers was shut down yesterday (April 23) and I think it was due to this or similar type of problem. Can you please, update on this?
I agree that CPU abuse should be discouraged. On my droplets I run a cryptographic research program that is somewhat related to Bitcoin (in that it explores certain characteristics of the secp256k1 ECDSA curve). This isn't Bitcoin mining (which - as pointed out - is completely useless on CPUs by now, even if you had a botnet - and even GPUs are best used for 'mining' so-called 'altcoins' and then trying to sell those for Bitcoin.. but that's a different discussion), but is also a high-CPU, low-disk/net usage process.
When I asked for additional droplets to be allowed ('only' 10, mind), I explicitly mentioned this along with the numbers: the process takes about 60% to 85% of CPU, depending on exactly which droplet I'm looking at. I don't know if that is a percentage of total core use, or of my droplet's allocated 'CPU slice' as it were.
However, if it turns out that my processes negatively impact other customers, I certainly have no issue with using e.g. cpulimit to artificially drop their CPU use and making some code changes to better distribute workloads ( I don't believe the regular schedulers would work, given that the running system has no idea about the outside systems and their demands. )
I think an alternative option - and perhaps that is in part what Moisey was alluding to - is to let users specify a CPU priority preference, where one system can dynamically be assigned less CPU time based on other systems' demands - thus giving plenty of CPU for the odd burst, while allowing processes such as mine and those already mentioned to continue to use the CPU to their full potential. Of course there's a good chance that those who simply want to hog the CPU as much as possible will set their preference thus - but those can then be dealt with specifically while letting 'good citizens' do their thing.
I'm looking forward to details about the CPU sharing changes, as well as input from others on how to deal with the general case of high-CPU-use processes.
I like Moisey's comment about the CPU sharing and how the allocation will change according to the amount of CPU used. However, clients should remember when purchasing a VPS is that they're getting it as OpenVZ - it isn't KVM or some other perfectly virtualized platform. CPU sharing is part of the system.
I'm glad DigitalOcean has been proactive on this.
Brandon Ouellette commented
I didn't realize that those droplets mining bitcoins could negatively impact our droplets, is this still an issue?
What does "mining digital currencies" mean? I know more or less what a digital currency is, but... what is happening?
So it's basically as I feared. Developers are going to lose burstable CPU capacity because of these miners.
Moisey - are you saying before that droplets were uncapped and would fight over CPU cores. If so, this would mean a dramatic loss of burstable CPU power for legitimate non coin mining users.
I feel that on the whole these miners are hurting all the customers who actually care about using your services to do development and host their sites.
The new sharing will allow the same exact behavior as before except for when a server has 0% idle CPU left in which case the droplet will be restricted according to the size of the droplet that was purchased.
While we are all for people developing around cryptocurrencies as a platform we don't feel that mining is best done in the cloud except in rare cases such as the immediate launch of a currency.
We will be rolling out the new share policy tomorrow on a few production systems to test for stability and see if there are any bugs and are hopeful that we will be able to roll this out across the cloud by Thursday.
If not then it will be rolled out Monday, as we try not to do major releases on Friday just before the weekend.
Just ban it in your ToS and people will think twice before exploiting your promotions with fleets of mining droplets. Let developers develop whatever, and encode all the videos they want. I'd like to hear that DO is protecting the interests of developers and not pandering to speculators and exploiters.
Can you elaborate about how the new CPU-sharing code will affect performance for your wider customer base?
We do not want to ban this activity outright because there are many developers that are interested in crypto currencies and there are some cool applications that can be written.
Anyone serious about mining however really stands to gain the most from a GPU based system, however when there is a new currency launched it can still be profitable to be the first to mine it even on a CPU based system.
As a result we are looking to implement a more fair share CPU sharing which will improve overall reliability in all cases. Additionally there are other use cases that have very similar usage patterns to bitcoin mining that aren't as prevalent to abuse such as video encoding.
We should have most of this ready by the end of next week but I'm sure there will be a couple of bugs that will pop up that we will need to sort through.
Zachary DuBois commented
Agree, +1. Needs to go against the TOS or something. Or make a special plan (A checkbox when deploying if you are going to demand high CPU) that will put you on a physical server with other high CPU customers. My Droplets take 25%-50% CPU depending on traffic and time of day.
Why don't you explicitly ban this activity in your Terms of Service? All it does is harm your core customers: developers and small business.
Agree very much with this. Some serious CPU hogging going on.
This should totally happen, I think it should really be considered as abuse of the service since CPU resources are shared.
Agree with this!
This HAS to be done, its crazy!
Everyone getting 2 months free mining, tonnes of tutorial sites saying they should do it.