Hydration oracle manipulation - post mortem
On the 3rd of July 2024 we have received a report in the Immunefi platform from top hacker on the platform labeled as “Critical” called “Risk free oracle manipulation”. and it looked as following:
Hydra is an interesting substrate based protocol that has lots of features. Essential to the functioning of hydra is the validity of its price oracle, which computes an exponential decay metric based on the historic prices of its own markets.
Two pallets in hydra make it feasible to perform a reliable oracle manipulation attack: utility and dca.
Various critical aspects of the system rely on the reliability of the price oracle. One example is the limitation of liquidity provisioning while the spot price is far removed from the oracle value. This protects the protocol from a simple attack where an attacker manipulates the price of an asset, to then add liquidity and reverse price manipulation with a much higher constant product.
In this report we’ll focus on the oracle manipulation attack, as the codebase indicates that the project developers are very familiar with the attacks possible without valid oracle prices.
Vulnerability
The interplay of the dca and utility pallets creates a situation where we can reliably manipulate an oracle without risk.
The attack components are as follows:
Using batch_call we can create two batches of extrinsics with such a weight that no other transactions will be included in this block.
Using dca we can craft a trade which will get included at the start of a block before anyone else can do anything.
The attack steps are as follows:
Create a dca transaction which sells a lot of token X, carefully configuring slippage.
Create a batch call that fills the block weight and buys a lot of token X
Repeat for multiple blocks
(1) only executes when (2) has successfully executed in the previous block (2) only executes when it fills the whole block
Cost of the attack: The attacker does incur some slippage while they’re running the attack. There is a dynamic fee that goes up to 5%. However, this is based on net volume, which is 0 for most of the attack.
The attacker also has to pay transaction fees, which don’t seem to be too high.
Limitations:
- (2) might not be included in every block
This doesn’t lead to a loss of funds for the attacker, as the price is at a non-manipulated state.
- Other people might setup dca while the attack occurs
There seem to be a few options for mitigating this problem, one of them would be to fill the available capacity for dca orders.
- The attacker can only manipulate the price each block by 50%
This limitation is within a block, in consecutive blocks the attacker can extend their attack to change the price more.
This is attack is similar to the TWAP oracle attacks possible when one controls multiple validators in the ETH network: TWAP Oracle Attacks: Easier Done than Said.
It then followed with proof of concept:
In the proof of concept I’ll explore the core mechanism of the attack:
Preventing others from acting while we set up the attack
Immediately closing the position using dca
We'll keep things simple and just buy a single ETH, focusing on the capability of changing the price and reversing that change without interference.
There are two files:
poc.jsandchopsticks.ymlprovided in the separate PoC field.To run the poc you’ll need to install
@polkadot/apiand@acala-network/chopsticks, start chopsticks with the provided configuration and run poc.js.The output when I run the poc:
alice : Current status: Ready alice : Current status: Broadcast bob : Current status: Ready bob : Current status: Broadcast Progressing to next block charlie : Current status: Ready charlie : Current status: Broadcast charlie : Current status: Invalid alice : Transaction included at block hash 0x63f70b9a52a752feeb5000d4b1a610aaebec1ded1c77599c3f55dd1e31cc0d2f alice : Transaction finalized at block hash 0x63f70b9a52a752feeb5000d4b1a610aaebec1ded1c77599c3f55dd1e31cc0d2f bob : Transaction included at block hash 0x63f70b9a52a752feeb5000d4b1a610aaebec1ded1c77599c3f55dd1e31cc0d2f bob : Transaction finalized at block hash 0x63f70b9a52a752feeb5000d4b1a610aaebec1ded1c77599c3f55dd1e31cc0d2f Progressing to next block Printing summary of balances through steps Step Init Alice: WETH: 10000000000000 HDX: 1000000000000000000 Bob: WETH: 10000000000000 HDX: 1000000000000000000 Charlie: WETH: 0 HDX: 1000000000000000000 Step first block Alice: WETH: 10000000000000 HDX: 999999999999999999 Bob: WETH: 10000000000000 HDX: 999999999999999999 Charlie: WETH: 0 HDX: 1000000000000000000 Step second block Alice: WETH: 10000000000000 HDX: 999999999999999999 Bob: WETH: 9999999999999 HDX: 999999999999999999 Charlie: WETH: 0 HDX: 1000000000000000000 result Alice: WETH: 0 HDX: -1 Bob: WETH: -1 HDX: -1 Charlie: WETH: 0 HDX:As you can see Charlies’ transaction was excluded while bob’s wasn’t. We’re good even if for some reason Bob’s transaction isn’t included, bc nothing would have happened. Even if Charlies transaction is included in the next block it wouldn’t do us any harm.
In this case we do see our DCA continuing. I’ve kept it like this so you can see that the dca transaction is running.
Proof of concept: https://gist.github.com/jak-pan/e49d74bbf45d41169a9599eba75d301e
NOTE: Gists were copied to preserve anonymity of whitehat upon his request
We have immediately started to look at this as a potential threat given the oracles are used in add and remove liquidity functions.
In a follow up message the whitehat explained how this flaw might be used to exploit a previously known vulnerability that was mitigated through various measures including the addition of liquidity addition and removal fees:
If I were to exploit this vulnerability, I would inflate the smaller pools, hold the price at an inflated point for a couple of minutes, then add liquidity while maintaining price difference for another few minutes to then sell everything I had.
In a normal pool:
buy X
add liquidity
sell X </aside>
We have understood that if the attack is possible in this scenario, it would require multiple blocks to execute, very sophisticated setup and huge amount of funds of different Omnipool assets. It would need to fill the DCA capacity while partially also filling the normal blockspace.
Nevertheless, we didn’t discount this, but started to do our own analysis of the pool DCA and fee structures. While we analyzed this we have pointed out that there are risks involved to the potential hacker:
Arbitrage bots
Token liquidity caps
Liquidity add and remove caps per block
Maximum price change limit per block
DCA price change limit per period
Randomized oracle execution in the block
Dynamic add and remove liquidity fees maxing at 1% before disabling these actions until price stabilizes
Dynamic volatility fees hitting up to 5% each manipulation block
Technical committee ability to pause and unpause Omnipool activity and XCM
The presence of all these factors had a significant impact on the complexity of the feasibility assessment of the attack. Due to this complexity, Immunefi was asked to provide an analysis and recommendation. In collaboration with the whitehat we’ve built various simulations and threat models. We believe there was no risk of loss of funds due to the excessive liquidity requirements and limitations listed above.
Although we believe that funds were never at risk, we’ve introduced extra limit on DCA trades which mitigates any potential risk from the vector reported by the whitehat. It is now impossible to perform.
While we were assessing the original submission, we have started to build our own “chopsticks” version of the attack, as well as internal threat model based on real value flows with full environment execution. In the final scenario, the attacker is trying to exploit oracle while also leaving the Omnipool state at manipulated price for the time of the attack. Causing significantly higher risk to the attacker if the attack could be intercepted and arbitraged.
This is the final POC we received after multiple iterations: https://gist.github.com/jak-pan/f977db4b897e595826a7a2865c591a95
The presented attack would look as following:
Preparation: Attacker would accumulate tokens in Hydration from the Omnipool and other venues to be able to add them later in the attack. This step could take weeks.
Manipulation: Attacker would start manipulating the price using DCA and batch call.
Batch call is used to modify the prices at the end of the block, this call is necessary to be last in the block, requiring filling large portion of the block with other transactions. This transaction will move the prices 50% upward which is maximum in block limit for price movement inside Omnipool.
DCA is scheduled in the same block to bring the price back the maximum 50% to the original price, giving no space for arbitrageurs to extract value from the attacker as it immediately hits the limit. Price is now at the original value from the previous block but the oracle was updated during the initialization phase with the increased price. The block is using ~10 block EMA prices and as such, attacker will need to sustain this for this time to move the price significantly.
Attacker can repeat the step A but since the price checks from this blocks are currently at -50% he can move the price with the batch call 100% in the opposite direction to hit the +50% limit. This would leave the pool at manipulated state, giving arbitrageurs the opportunity to extract value from the attacker. However he schedules DCA which is executed first in the next block to go back 50%. We could assume this would be very hard to intercept if arbitrageurs don’t use this technique. This is repeated until the prices in Omnipool are manipulated at ~7* as shown in the POC. During 14 blocks.
Stabilization: Once the prices are inflated, the attacker needs to continue the price manipulation at a flat price (2.a.) for the volatility checks to stabilize and to be able to start adding liquidity. During this period the attacker is incurring dynamic fees which will slowly decay.
Liquidity addition: After the prices in the oracle stabilize, the attacker will add liquidity to the pools that are attacked keeping in check the ratios of the pools and per block limits on add liquidity which is 5% of existing liquidity per block.
Return to mean: The prices are moved back to original state with DCAs by maximum of 50% per block.
Stabilization: The price needs to be manipulated as in 2.a to stabilize the oracle and allow for withdrawals
Withdrawal: The liquidity is withdrawn with the limit of 5% of the existing pool size per block.
At this point, the POC seems to show value extraction, but the simulations that were provided to us have circumvented some of these limitations, and were working on unrealistic Omnipool setup. These are required for Critical level payout by our bounty programme.
Final attack breakdown
The setup is using a price inflation stage which almost doubles the Omnipool TVL for DOT token to simulate attack on the whole Omnipool without considering other factors.
The attacker in the POC assumes he will be able to perform perfect arbitrage of the tokens between the different assets in the pool. While keeping the other actors at bay.
If one token is swapped for another token during this attack (USDT for DOT), both tokens are now arbitrage-able by any other token (and actor) in the pool. If attacker uses USDT to increase price of DOT, the USDT price drops compared to other assets, which allows arbitrageurs to step in, while also making it harder for the attacker to perform the attack on such token, since he moved the price in the opposite than desired direction. This itself requires perfect balancing of the pool and having inventory of significant size consisting of all of the tokens in the Omnipool. We don’t believe this is possible to maintain as the attack method breaks this balance in its principle.
We believe this alone makes the attack non-profitable, however let’s breakdown the attack steps as shown previously together with countermeasures present:
Preparation: Since the attacker needs to gather significant amount of assets from the ecosystem many of which are only available in the Omnipool, the attacker is incurring significant risk. The attacker (as per this POC) would need to gather roughly 50$ million of various tokens, while some of them are easy to get, some of them are impossible to gather in quantities required to keep the Omnipool balanced as it is the only venue to buy these tokens.
Manipulation: The attack is not using proper execution in the POC, thus circumventing some of the dynamic fees, DCA execution limits and assuming full on success without arbitrageurs involved.
It also fails to follow the limit of maximum price change during the period of ~10 block short oracle, which is bound by type to 100% during schedule creation. This translates to either roughly 16% of the liquidity being open to normal in-block arbitrage during each block in the price manipulation stage, part of the stabilisation stage, and the sell-off stage. We must assume at least part of this opportunity will be taken by arbitrage bots. As a result of using these limits the price change during the same time period is roughly 4.5x, without counting in the arbitrage making the viable attack longer, requiring considerably higher amount of liquidity, while being exposed to considerably higher risk.
This could be circumvented by using advanced stepping technique where attacker moves the price more slowly on the way up and then subsequently down so that it is not exposed to arbitrage. However, this prolongs the attack and incurs additional fees. This is also where we implemented additional limit to prevent the attack completely.
Stabilization: At this point the attacker will incur ~30% hit on fees of the total amount used to manipulate the price. This step will incur additional fees which are slowly dropping from 5% per step to less than 1% over the period of multiple blocks to allow the next step to work.
Liquidity addition: The liquidity add is capped at 5% of the amount of the respective token, since we have moved the price and the liquidity is now small, this would take more time to complete.
Return to mean: The attacker incurs another fee hit every block on the price movement back to original state as the fee movement is not symmetric when the price moves up or down and is applied mainly to token leaving the pool, attacker will now suffer another loss in the opposite direction while being again open to arbitrage opportunity in the normal blockspace.
Stabilization: To stabilize the fees and volatility triggers, attacker needs to maintain the attack at the steady price to be able to withdraw, incurring additional fees before he can withdraw.
Withdrawal: Withdrawal of the funds is capped at 5% of the liquidity of the given token per block slowing the potential attacker further.
The capital requirement for manipulating DOT pool we have from the POC itself is roughly 0.7 final manipulated liquidity of the Token + 30% fees + 0.5 final manipulated liquidity of the manipulated token + 30% fees.
This translates to at least 30million$ of liquidity for manipulating the DOT price to levels as shown in the POCs, while losing significant portion of the capital to the fees and discounting acquiring and holding other tokens in the Omnipool in preparation for the attack. It would be safe to assume that the liquidity required to perform this attack is multiple times higher than the liquidity in the Omnipool.
Although it was possible to manipulate the price oracle, our simulations shown it would be infeasible to leverage this to do so profitably. We however understand, that this is a critical part of our protocol and could become an issue in the future.
For this reason we’ve decided to offer the whitehat highest payout for High tier of 25k$ which matches the lowest reward of Critical payout.
Over the last 6 months, as a result of this report, we have learned and improved a lot. We have built internal threat modelling and monitoring tooling and adjusted protocol parameters to protect from similar attacks in the future. At the same time, we will be updating the rules of our bounty programme based on our findings of feasiblity for multi-block attacks to streamline our triaging process in the future.
We want to express that we greatly value the time and the effort of everybody included including Immunefi and the whitehat, as he found and reported a bug that could become a serious issue in the future and would like to express gratitude towards anybody trying to make Hydration more secure and decentralized.
