Cryptocurrency Price Scrapping using BASH and Coingecko API
Cryptocurrency is in such a hype that it is attracting even nerds and terminal fanboys, it is quite comfortable for such people to view the price inside of thei#bash
Cryptocurrency is in such a hype that it is attracting even nerds and terminal fanboys, it is quite comfortable for such people to view the price inside of their terminal, also for normal people to learn about how to interact with an API from the local machine. We are gonna make a script about 20 lines in BASH to extract data from the coingecko cryptocurrency API and some tools such as grep and sed. That being said let's start scripting.
This is how the script will work like. It is not a fancy script just some basic commands to extract data from the API.
Getting familiar with the API
This is quite an important step as this will decide what type of data we will get from it. So head out to coingecko cryptocurrency API and start exploring some things yourselves. I am gonna cover everything you need for the script but just for your knowledge and if you are really into crypto.
Hopefully, you should see the following screen and after clicking on "Get Started", you will see a list of the API endpoints. After playing with it for several hours, I found the perfect and precise endpoint to achieve our target. Many o them work well but the problem with many of them was the precision of the price, there it could not show a reliable price for some low-valued coin/tokens. There are tons of options to choose from, but every option has its own advantage and disadvantage.
Based on my tiny experience, the best fit for the required task will be the "Get historical market data to include price, market cap, and 24h volume (granularity auto)" option in the coin section. It will give the price with very nice precision and also in any national currency as well. So when you find the option, which looks like this:
If you click on that option you will get a button to "Try it out" , just click on it and now you can parse data to the API and respond with the JSON response file like the text below as output.
Filling in the coin id
You need the
which is the code of the cryptocurrency on coingecko so a few ids of famous cryptocurrencies are:
, and if you want more coins you can check the coingecko API coins list
and now you can search using the shortcut F3 and type the name of the coin you require.
Filling in the currency
We need a currency to display the price of the selected coin in terms of a particular currency or its equivalent price in the currency. Every national currency has a code attached to it, for example, USD is the code for US Dollar, INR for Indian Rupee, EUR for Euro, AUD for Australian Dollar, and so on. You can find the list here .
The number of days
We also need the number of days to indicate the price of the coin before the provided day, but we want today's and latest price, so we can enter the number of days as 0. You can get the price as your wish but we want the current price so we are using 0.
So your request should look somewhat like this, just for particular coin pair:
After filling in with your favorite coin-currency pair, you can now click on "Execute" . You will get a list of prices, volume, market cap, and all of that stuff. It should look somewhat like this:
If you look carefully and are aware of the current price, the second item in the prices object is the actual price of ethereum at some point in time. We need to extract that using tools and utilities in BASH. They have also provided a cURL command to fetch the API, how nice of Coingecko :)
So, that was the introduction to the Coingecko API, you can now explore various endpoints and what they respond to, and what are inputs they take. Many of them do not give precise enough price, so check out that as well.
Fetching data from the API
Now, we'll finally start coding. Firstly we'll need input from the user, the coin-id, currency code, and optionally the number of days. We will read the input by using the read command and providing the prompt argument.
#!/bin/bash read -p "Enter the coin name: " coin read -p "Enter your national currency: " crncy days=0 #read -p "Enter the number of days before today to get its price: " days
Now we need the golden piece of this script, the cURL command. The command is just pre-written for us XD. But hey we need to modify it a bit. We need to store the output of the cURL command in a fie for modifying it further. We can use piping the command and avoid using files but that would make the command quite big. So, I just use the files to handle and trim the output according to what I need.
This is the default command that coingecko gave us:
#!/bin/bash curl -X 'GET' \ 'https://api.coingecko.com/api/v3/coins/ethereum/market_chart?vs_currency=usd&days=0' \ -H 'accept: application/json'
We'll change the default command to this,
curl -o temp.json -X 'GET' \ 'https://api.coingecko.com/api/v3/coins/'$coin'/market_chart?vs_currency='$crncy'&days='$days'' \ -H 'accept: application/json' &> /dev/null
'$variable'in between the URL to embed the variable value in it. We change the ethereum or any coin name with
'$coin'and the currency name with
'$crncy'and the same for the days as well. We have to store the output in the temp.json file, we use -o to output the result in the specified file in the cURL command. It's optional to add
&> /dev/nullbecause it just flushes the output of cURL, it looks neater if we add it.
And that is it, we have obtained the JSON response and stored it in a file, we have the access to the data and we just need to edit the file.
Editing the response JSON file
Now we start the actual editing and pattern finding in BASH. We have a plethora of tools to use like grep, sed, awk, and others, but I'll use only grep to keep things simple. Firstly we need to see the output( JSON file) again.
But the file stored locally has everything in a single line, like this:
So, we just need the numbers between
The text between
It is quite simple to extract the required value, we will use grep and Pearl Regular Expression.
grep -o -P '(?<=,).*(?=]],"m)' temp.json > price.txt
]],"mfrom the temp.json file and storing the output in the price.txt file. As simple to use and we have the current price of the coin in terms of the provided currency in the file price.txt.
Now we have obtained the result in a crystal clear way, we need to store the price in a variable just for further usage and simplicity. We'll use a while loop that iterates over the file until it is End of the File. We'll store the value in the variable in the following way:
while read val do p=$val done < price.txt
pvariable. But we are not done yet, because if we see some values of certain coins which have quite low value, it displays in the scientific format. We'll tackle this in the next section.
Converting the price from scientific notation to decimal
If you try to print the values of coins with pretty low value like
or any other coin with less value then a penny. The value is expressed in scientific notation i.e like
This is not good-looking, is it?
price=`printf "%.15f" $p`
pvariable with a precision of 15 decimal values, that is enough for any serious small value coin.
Now that was much better. That is it! How simple was that. Really BASH has some powerful commands and tools.
Printing the price
Finally, we need to print the output, and to keep it simple, we can print using the echo command.
echo "The value of $coin in $crncy is = $price"
#!/bin/bash read -p "Enter the coin name : " coin read -p "Enter your national currency : " crncy days=0 #read -p "Enter the number of days past today: " days touch temp.json price.txt curl -o temp.json -X 'GET' \ 'https://api.coingecko.com/api/v3/coins/'$coin'/market_chart?vs_currency='$crncy'&days='$days'' \ -H 'accept: application/json' &> /dev/null grep -o -P '(?<=,).*(?=]],"m)' temp.json > price.txt while read val do p=$val done < price.txt price=`printf "%.15f" $p` echo "The value of $coin in $crncy is = $price" rm temp.json
That was it, I hope you liked it and learned something along with me. BASH truly has the potential to do a whole lot of stuff. This was just one of the many examples where we can do some data gathering or scraping from the web using various tools and utilities. Thank you for listening to me. Happy Coding:)