https://www.meetgor.com Tech blog by Meet Gor en-us Sat, 09 May 2026 05:38:52 UTC https://www.meetgor.com/posts/flight-observatory-mumbai-airspace-case-study A closer analysis on the airspace around Mumbai's BOM/VABB (CSMIA) Airport. Questioning the intuitions from memories and answering and reasoning through the data. Thu, 02 Apr 2026 00:00:00 UTC ## Introduction I do write blogs, newsletters, but this is something wild I had written in a while. I was always curious and interested in knowing about the flights flowing over Mumbai, I do always look up and see the Airline when it appears above my head in front or on the side. But I never actually read or discovered anything true or factual about it. As a software developer, that is very unsettling, I have some ideas and observations but never tested them. I can excuse myself that I am busy and that is a toy little project to get off my hands. But now, we have LLMs and no excuses. I don't need a second invitation to explore my curiosity, off I went. And came back with around a week or 2 of digging in data, downloading the ADS-B stream of airspace and crunching the numbers for the BOM (VABB) Mumbai's Chatrapati Shivaji International Airport (CSMIA) and its vicinity. The numbers I must tell you were staggering and I couldn't stop starring at them, I felt relieved and felt my childhood was worth while. I am writing this blog post to explain to myself, how I got here. It was a chaos in start, total misery of data. But after a few attempts of successfully accumulating the data only for BOM airport's vicinity of 300 kilometers, I was able to finally make sense of the pile of data (since at one point it looked like a goble of mess). This will be a two part blog. This one being non-technical (that is no grilling from peer developers) which will explain the why and what of the case study of the Mumbai Airport. The next part (out in a couple of days from now) will be a treat for developers (or curse words for myself, ok it won't be that bad), which will explain the how. This part is about my intuitions and the memories I had since growing up near the vicinity of Mumbai Airport. I explain what dataset I used (it was not a simple available dataset in the public, mind you that), how I structured it, what questions I asked, how I formed the queries (not in terms of SQL but human logical terms), and how I reason through the results. And trust me everything made sense. So, if you have been in Mumbai, and wondered how many planes flew above your head in a day, you are in for a treat! Buckle up! Because Mumbai Airport never stands still. Check out the full case study here -> [Flight Observatory - Mumbai Airspace Case Study](https://dev.meetgor.com/flight-observatory/case-study/mumbai-airport/) ## Flashback of Mumbai from 2010-2018 I have spent my life in Mumbai, the two decades that I have lived here, I often feel nostalgic lookin back at the memories. The one that stands out even today is the plane spotting. Since childhood, I have seen, heard, observed planes flying above me, beyond the buildings, buzzing and slashing the concrete jungles and flat blue slums. I distinctly remember the days from my school, the 7th A classroom, located on the top of the school, the highest room on the school. I remember the days I shared sighting planes through the periods with my friends. It was a nostalgic experience to see - the regular "Air India" on time, - "Lufthansa" with its guzzling sound, - occasional "UPS" with the jumbo cargo, - and the millions of "Jet Air" and "Spicejet" among them. There was a different joy of just seeing and observing them, admiring the beauty of Boeing 787, 777 and 747s, Airbus A330 and occasional very rare Airbus A380 the king. Those are fresh in my mind. I was lucky to be honest to have an airplane enthusiastic friends with me. One of them was notoriously a die-hard fan of British Airways though. The 11:45 British Airways was a treat to him, with me cursing words at British people, hoping the plane to crash (just teasing him of course). We almost had a anticipation on the next plane to come and over the year we had a schedule that which one will come next depending on the period at the day. We had formulated that the BOM Airport's frequency was 2:30 minutes (that's 24 flights in an hour, 576 in a day). That is a lot of flights landing over our head, I used to brag that BOM was (is?) the single runway busiest airport. And take a strange pride in defaming British Airways. Those were the times :) And now, I don't have that luxury to sit with that friend and gaze at the sky (if I will, I might get existential crisis I think). I wanted to get those moments back, I don't have the luxury to sit with that friend though, but I realize I have another friend a tool of sorts. LLMs. I realize I am a programmer now, whatever I want to do, I can. Its just a wish away. ## The Idea And I began making this project. To observe the sky. Not by my eyes. By tracking, analyzing and crunching data from APIs. Nothing comes for free. Constraints start developing, but there is no engineer who has no constraints. For me I didn't want to setup a cloud infrastructure(not that I can't I just don't have the budget for it) and do bunch of credit card swipes, I didn't have the luxury for that, and for buying heavy 200$ subscription of those AI-tools either. So what? > Engineers aren't made from freedom. They're shaped by constraints. I explored every API out there to see what I can use the best. And lo and behold I found some. Not everything was great, there were samples, just a day for a month, limited data, but I kept digging and digging. I found what I needed, data points for Mumbai (or any airport for that matter) across a decade. I thought I might get full-day, full-year data, a decade worth of data. But what I got was a decade worth of data, for all months, but only for the first day of the month. Sigh! But I kept moving. I don't care how little the data is. It is real. It is 12 days of the year for 8 years. That is not a lot, but enough to find patterns. And I can surely say, Mumbai airport hasn't changed drastically over the decade. I thought that was not enough! I wanted more, I wanted live data. Can I get live? I though I could, I did but sort of. Its not the most accurate representation of flights. But a good guess, a practical one at least. So, I built [Flight Observatory](https://dev.meetgor.com/flight-observatory). A live and historical data analysis tool or a webapp of sorts for viewing flights captured by the ADS-Bs around the world. I kept storing the live data and gobbled them into a historical section. From the other dataset and some live observations(which turned out to be a disappointment) I also made a case study for Mumbai Airport. This will be not quite nerdy and technical, even if you are not a technical person, strap on. Its going to be a fun read. The next part will be a treat for the technical people reading this, so put on your glazing glasses and bickering comments for that post later. ## What I found (TLDR) Mumbai Airspace is very consistent. It is busy in a very Mumbai way: a lot of movement, but with a clear pattern. The traffic pattern is strongest in the early morning and evening. The daily activity spreads across the early morning, the workday, and the evening. In the airport-proximate analysis: - The strongest hourly movement shows up around 07:00 IST - Other heavy hours around 06:00, 11:00, 17:00, 20:00, 21:00, and 22:00. On the sampled days, the data produced: - `21,923` airport-proximate events - Split into: - `12,723` landings - `9,200` takeoffs - With a median landing-to-next-takeoff gap of `1.5` minutes. That last number matters. It is the airport's strength and the health checkup number. It says the airport is turning movements over very quickly, which matches the reputation Mumbai has for being tightly packed. The airspace itself is occupied for a while: - Median landing run: `34.5 minutes` - Median takeoff run: `25.1 minutes` So landings stay visible longer than takeoffs. That makes sense operationally, arrivals spend more time in descent and approach, while departures clear out faster once they lift off. The same aircraft also comes back around fairly quickly: - median same-aircraft ground time: `96.5 minutes` That means the aircraft are not sitting idle for long. The fleet is being reused in a steady loop, which fits a high-frequency airport like Mumbai. And the shape of the traffic is not random: - Saturday is the busiest weekday in the sample - the landing and takeoff corridors split cleanly across the airport area - the dominant heading clusters show the same directional axis being used again and again Across the full analysis set, I ended up with about - ~17 million observations (that looks like a gigantic lot, but it isn't) - ~151,000 inferred flight runs - Spread across ~4,900 aircraft hexes - ~4,800 registrations - ~140 aircraft types. That feels right for Mumbai. It is a city that never really stops moving (it actually sleeps, but internationally awake), and the airspace behaves the same way. The airport-proximate movement counts are also not tiny, it is structured, dense, and repetitive. The airport works like a tight system, not a loose one. ## Notes and Clarifications The data itself was not clean in the neat, airline-report sense. It is raw observation data: points every few seconds, over and over again, with altitude, speed, heading, position, and callsign changing as aircraft move across the sky every 5 seconds or 30 seconds(depending on the year). I had to group those observations into inferred flight runs before it became useful. Once I did that, the picture got much clearer. Most of the useful traffic is post-covid traffic too. The analysis is heavily weighted toward 2022 to 2025. > The ADS-B for the BOM (VABB) actually started the tracking of airports around late 2017, early 2018, and that too was per minute. So it was very sparse and not necessarily capturing the full spectrum. > This data is from 2018 to 2026, not the full day of each month. It is sampled to be the first full day of each month (i.e. 1st April 2024, 1st of May 2024, and so on). Also the data points from 2018 to mid-2020 is snapshot of 60 seconds and moving from late 2020 through 2026 (till march) is snapshot of 5 seconds. > The data is only for ~78 days(not continuous, its the 1st day of month). 2022-2025 full years (12 days), there are only 3 days in 2018, since ADS-B was active after mid-2018, and the dip in 2020-21 due to COVID, so little or no traffic observations. ``` ---------------------------- | Year | Number of days | ---------------------------| | 2018 | 3 | | 2019 | 11 | | 2020 | 5 | | 2021 | 9 | | 2022 | 12 | | 2023 | 12 | | 2024 | 12 | | 2025 | 12 | | 2026 | 3 | ---------------------------- ``` > The numbers are real, but they are still a proxy. They are not ATC logs. They are not airport ops records. They are ADS-B-derived movement patterns around Mumbai. That means the data is powerful for showing pattern, density, clustering, and dominance, but it should not be mistaken for official runway statistics. This data doesn't infer the activity the airport is doing exactly as in an official ATC sense, but it does tell you something real about how packed and continuous the traffic feels once you look at it through ADS-B data. > ADS-B short for Automatic Dependent Surveillance Broadcast that allows the aircraft to capture its own position and other details like (altitude, GPS based location, direction, speed, hex code i.e. unique aircraft code) and broadcasts (sends in the open as a radio signal), this gets captured by listening Air Traffic Controllers(ATC), Airports, satellites, networks, etc which then track and use it for internal purposes. That could be then used for tracking on the Airport, communnicating, averting accidents, etc. ## Initial Questions and Exploration I was looking for a historical dataset that could show me flight activity over Mumbai. I did not expect it to be perfect, but I did expect it to be workable. The first thing to understand is that this is not a clean flight log. It is a stream of observations. Each aircraft keeps broadcasting its position, altitude, speed, heading, and identity over time. That means the raw data is not `one row per flight`. It is more like `one row per movement in time for a given aircraft`. To make it useful, I had to turn those repeated observations into inferred flight runs. That is what makes the analysis readable. Instead of staring at thousands of raw position points, I can ask more human questions: - When is Mumbai busiest? - Which airlines show up the most? - Which aircraft dominate? - Where do planes start descending? - Where do they take off from? - How long is the gap between one landing and the next takeoff? That shift from raw observations to inferred runs is what made the rest of the project possible. I landed on the ADS-B Exchange historical data because it looked promising. At first glance, it seemed like I had found a long historical archive. But there was a catch I did not fully appreciate at the beginning: much of the data was sampled by month, and a lot of it was only the first day of each month. That was smaller than I hoped, but still useful enough to extract patterns from. That was the point where I stopped thinking about the data as a raw archive and started thinking about it as something I could actually analyze. ## Making "sense" of the data So, I had millions of observations. Because a single minute has 12 snapshots and if you do the math (not asking you to do it, but if you are curious), we end up with around 17,280 snapshot for a single day (12 snapshots per minute * 60 minutes * 24 hours). If you stack it up for around 12 months, that really grows big. So, what exactly were these snapshots? Could I directly use them in querying the database? No, not quite. It actually depends on what questions we might need answering. ### What is a Snapshot here A snapshot is basically a list of observations (or set of status) for the aircrafts currently in the airspace. The initial snapshot that I download from [ADS-B Exchange](https://samples.adsbexchange.com/index.html#readsb-hist) has a full world air-traffic and I never want that full data, I just want the tiny airspace around the Mumbai one. So we filter the latitude and longitude position around Mumbai. So, each snapshot finally becomes the aircrafts flying around the radius (30km to give a number) at that time. So we take those kind of snapshots at 5 second interval. So let's suppose there are 2 flights around Mumbai currently one landing and one taking off: These will be a snapshot. ```json [ { "icao24": "123456", "latitude": "19.123", "longitude": "73.123", "altitude": 500, "velocity": -100, "heading": 90, "timestamp": "2022-01-01T00:00:00Z" }, { "icao24": "8901", "latitude": "19.083", "longitude": "72.123", "altitude": 1000, "velocity": 300, "heading": 90, "timestamp": "2022-01-01T00:00:00Z" } ] ``` After 5 seconds you might see a change in the latitude and longitude (positions), the speed, and the altitude (also the timestamp). So we get another snapshot which shows the status of those aircraft currently in the radius of Mumbai, maybe in the next snapshot another aircraft is about to land, so it appears in the snapshot too. So on, you get the point right? If the aircraft stays within the radius of Mumbai for more than a minute (which everyone might, often more than a minute even 10-20 minutes?) then we might have 12 observations for that aircraft across those snapshots. We basically have a record or a progression of the aircraft over time. That is, in essence, a time series data. We are measuring a change of an entity over a period of time. So, this is what I get from the downloaded snapshots which are in millions. How do I make sense of them? Well, let's ask a question, then the scope can be reduced or we can move in specific direction. ## Questions on the data > What are the total number of flights over Mumbai in 2026 so far? Well, now you can see, we are talking about flights or distinct aircraft over a time. That is a distinct thing from the observations we have, we have to first lock in the time, but the deeper question is how do we group these observations to get flights? That is why the question need to change a little before it can be answered properly. I am asking how many distinct flight runs exist in the data, how long they last, when they start, when they end, and whether they are landing, taking off, or just passing through the Mumbai airspace. Once I do that, the raw snapshots stop being noise and start becoming something I can actually reason about. So, what constitutes a flight run? That is a question that we need to answer before proceeding. #### What is a Flight Run A flight run is my way of saying: this group of observations belongs to the same aircraft movement. It is not one row, and it is not one timestamp. It is the whole stretch of time where the same aircraft keeps showing up in the snapshots and its position, altitude, speed, and heading keep changing in a way that looks like one continuous movement. Now, how do we distinguish an aircraft? A unique aircraft means one distinct plane in the dataset, usually identified by its unique aircraft code, like hex or registration. This matters because the same aircraft can appear in many snapshots while it is in Mumbai's airspace. If I only counted rows, I would be counting the same aircraft again and again. But if I group those rows into a single run, then I can treat it more like one flight-shaped event. That gives me something much closer to what I actually want to study. ### Observations to Flight Runs So, I loaded the raw snapshots/observations into the dataset after compressing from json to csv to parquet files (dont worry about the details), into a duck database session. I had table schema like: - `observations` is just all the monthly Parquet files unioned together. - `observations_clean` cleans the fields and turns text like altitude, speed, lat/lon, etc. into usable numeric columns. - `observations_enriched` adds convenience fields like registration, aircraft type, airline_code, is_indian_registration, etc. From this we need to group into a flight run that could track each flight. So, I went and created a view that will do that: As we know we have `hex` in the data for each observations, that can be used to identify unique aircrafts in the airspace, however there is a catch. Each aircraft in ADS-B data is identified by a unique hex code, which represents the physical plane. But planes don't operate just one journey, they fly multiple routes throughout the day. This is where the callsign becomes important. A callsign represents the flight the aircraft is currently operating (for example, AI101 or 6E203). So, we need a combination of `hex` and `callsign` to identify a flight run. I sorted the observations by aircraft and time, and then compared each row with the one just before it. If the same aircraft disappeared for too long, I treated that as a break. If the flight code changed, I treated that as a new run too. That gave me a cleaner way to read the data. The same aircraft could still appear many times, but those repeated sightings would stay inside one continuous stretch instead of being counted as separate things. In other words, I was no longer looking at a pile of random snapshots. I was looking at movement. Once that rule was in place, the rest of the questions started making sense. I could count runs instead of rows, compare landings and takeoffs, look at how long an aircraft stayed in the airspace, and see when the traffic was building up or dropping off. That was the point where the data stopped feeling messy and started feeling usable. That is the first step to answer the question, how many flights are there in Mumbai over 2026? ### How many flights are there in Mumbai over 2026? Once I started grouping those repeated sightings into flight runs, the question became much clearer. Instead of asking how many rows exist, I could ask how many distinct aircraft movements exist from 1st of January, February and March of 2026. That is the number I actually care about. It tells me how much traffic was visible, when it showed up, and how those movements were spread across the day. There are approximately `7,237` inferred flight runs for the 3 full days of the 2026. For the 12 full days of 2024, there were approximately `51,441` inferred flight runs, and for 2025 there were approximately `43,324`. For 2026, it is just the first quarter (3 days compared to 12 days of those earlier years, so a good balance). Now we can move into interesting questions ### When is Mumbai most busy? Once I had the raw snapshots grouped into flight runs and filtered down to the airport-proximate ones, the first thing I wanted to see was simple: when does Mumbai actually get busy? For a "24 hour day" chart, I used: - one row per hour - count movements for each sampled day + hour - then average those hourly counts across sampled days That gives us a number representing on a typical sampled day, how many movements happen at 07:00, 08:00, etc.?  The answer is pretty clear. In this sample, the busiest hour is 07:00 IST, with 1,086 movements. Right behind it are 20:00 (1,074), 17:00 (1,073), 18:00 (1,069), 21:00 (1,044), 11:00 (1,043), 19:00 (1,038), and 22:00 (1,032). The quietest hour is 03:00, with just 246 movements. So the traffic is not flat at all. It has a shape, and that shape is very visible once you count the flights as runs instead of raw rows. However if we take the average across all the sampled days, we get the same picture. For construcing the query for getting this, we want to know when the airport is most active during the day, so we can: - Keeps only airport-proximate events (lat and lon) - Labels each event as landing or takeoff (based on speed and altitude) - Converts the time to IST - Groups by hour of day - Counts movements in each hour bucket  If we take the median instead, then the busiest hours are 17:00 IST, with 30.46 movements, 18:00 IST with 30.54 movements and 07:00 IST with 30.17 movements that makes it very consistent to the average. The quietest hour is still 03:00 with just 7.69 movements. That gives us a good answer and the difference is honestly not that big. The busiest hour is 07:00 and 17:00 with summed average and median respectively, and quietest is 03:00. The summed view and the averaged view tell the same story, but from two slightly different angles. In the total sample, 07:00 is the busiest hour, followed closely by 20:00 and 17:00. In the average-per-day view, the evening shifts slightly ahead, with 17:00 and 18:00 taking the top two spots and 07:00 still staying near the top. The quietest hour is the same in both views: 03:00. So the exact peak changes a little, but the shape does not. Mumbai is still busiest in the morning and evening, and quietest in the late night. ### Which Airlines Show Up Most I want to know which airlines dominate the traffic. Once the rows were grouped into flight runs, I could stop looking at raw sightings and just count which airline codes kept appearing. I used the first part of the flight code as a rough airline label and counted the runs for each one. That makes the dominant carriers obvious instead of buried inside repeated observations. What I did was with a query: - Takes the grouped flight runs - Pulls the airline prefix from the flight code - Counts how many runs each prefix appears in What we get: - A dominance chart - It shows which carriers keep repeating in the sample - IndiGo and Air India should stand out immediately This was quite evident, as I have observed Mumbai's sky for a long. Indigo and Air India are firmly dominant across the board. However, there are some other interesting observations like Vistara Air, Akasa and Emirates. In earlier pre-covid era, it used to be Jet Airways, but now Indigo has taken the lead. The difference in the first 3 is quite a big one, and I think its right! Since Indigo and Air-India, not only in Mumbai, they are preferred over other domestic airlines. Some other notable ones down the order are `SIA` for Singapore Airlines, `DLH` as Lufthansa, and `BAW` as British Airways. They are not as heavily appearing as domestic ones, they are regulars on a routine, each day every day. ``` ┌──────────────┬───────────┐ │ airline_code │ movements │ │ varchar │ int64 │ ├──────────────┼───────────┤ │ IGO │ 55050 │ │ AIC │ 20892 │ │ VTI │ 9789 │ │ AKJ │ 6923 │ │ UAE │ 5057 │ │ SEJ │ 4869 │ │ AXB │ 4514 │ │ QTR │ 3728 │ │ GOW │ 2191 │ │ ETD │ 2186 │ │ IAD │ 2151 │ │ @@@ │ 1708 │ │ SVA │ 1660 │ │ ABY │ 1542 │ │ ETH │ 1504 │ │ OMA │ 1479 │ │ SDG │ 1448 │ │ GFA │ 1214 │ │ SIA │ 979 │ │ KAC │ 825 │ │ DLH │ 771 │ │ THY │ 754 │ │ MAS │ 745 │ │ BAW │ 721 │ │ FDB │ 705 │ └──────────────┴───────────┘ ```  A good validation query! It seems the data was correctly analysed. ### Which Aircraft Type Dominates This was something I didn't have much idea about. I was not well versed with flight models and their names. So I let it shape what we can find. ``` ┌────────────────────────────────────────────────┬───────────┐ │ aircraft_type │ movements │ │ varchar │ int64 │ ├────────────────────────────────────────────────┼───────────┤ │ A20N (Airbus A320neo) │ 49269 │ │ A21N (Airbus A321neo) │ 27739 │ │ B38M (Boeing 737 MAX 8) │ 13138 │ │ A320 (Airbus A320) │ 12587 │ │ B77W (Boeing 777-300ER) │ 7543 │ │ B738 (Boeing 737-800) │ 5643 │ │ B789 (Boeing 787-9 Dreamliner) │ 3162 │ │ A321 (Airbus A321) │ 2761 │ │ B788 (Boeing 787-8 Dreamliner) │ 2482 │ │ B77L (Boeing 777-200LR / 777 freighter family) │ 2428 │ │ A359 (Airbus A350-900) │ 2385 │ │ A333 (Airbus A333) │ 2067 │ │ AT76 (A319) │ 1862 │ │ A319 (Airbus A319) │ 1736 │ │ A388 (Airbus A388) │ 1648 │ └────────────────────────────────────────────────┴───────────┘ ```  ### Where Landings Start The question here is not where the aircrafts lands on the runway. It is where the aircrafts start coming down. That is a different thing. If I want to find the landing corridor, I should not look for the final stop point. I should look for the first clear movement in each run when the altitude starts dropping in a way that looks like descent. That gives me the start of the landing path, which is what matters for the map. Once I have those descent-start points, the next step is simple: group them by location and altitude. If the same area keeps showing up, that means aircrafts are repeatedly beginning their descent there. That is how the east-side corridor and the Ghatkopar-area low altitudes start making sense. The map is not showing a single landing dot. It is showing the place where arrival begins to feel like arrival. The patterns for a sharp dip is observed near the airport only, but overall, the gradual dip is seen on the Navi Mumbai side which is very true. The aircrafts gradually lowering their speeds and altitude descending into the runway strip. - 19.09,73.02 is east of the airport, out toward the eastern Mumbai / Thane creek / Navi Mumbai corridor. - 19.09,73.08 and 19.09,73.10 are farther east, so they sit deeper into the Navi Mumbai / Airoli / Ghansoli / Thane-side approach corridor. ``` ┌────────────┬────────────┬────────────────┬────────────────────┬───────────────┐ │ lat_bucket │ lon_bucket │ descent_points │ avg_alt_ft │ median_alt_ft │ │ double │ double │ int64 │ double │ double │ ├────────────┼────────────┼────────────────┼────────────────────┼───────────────┤ │ 19.09 │ 73.08 │ 142 │ 3405.6338028169016 │ 3400.0 │ │ 19.09 │ 73.1 │ 135 │ 3652.962962962963 │ 3600.0 │ │ 19.09 │ 72.86 │ 133 │ 36.2406015037594 │ 0.0 │ │ 19.09 │ 73.02 │ 123 │ 2638.008130081301 │ 2625.0 │ │ 18.58 │ 73.94 │ 122 │ 2115.1639344262294 │ 2175.0 │ │ 19.09 │ 73.01 │ 119 │ 2439.285714285714 │ 2425.0 │ │ 19.09 │ 73.11 │ 118 │ 3747.4576271186443 │ 3750.0 │ │ 19.09 │ 73.09 │ 116 │ 3565.0862068965516 │ 3525.0 │ │ 19.09 │ 72.94 │ 113 │ 1342.9203539823009 │ 1350.0 │ │ 19.09 │ 73.12 │ 113 │ 3951.7699115044247 │ 3925.0 │ │ 19.09 │ 73.23 │ 108 │ 5678.7037037037035 │ 5675.0 │ │ 19.09 │ 73.24 │ 102 │ 5867.156862745098 │ 5850.0 │ │ 19.09 │ 73.13 │ 101 │ 4157.425742574257 │ 4175.0 │ │ 19.09 │ 73.07 │ 99 │ 3325.5050505050503 │ 3325.0 │ │ 19.09 │ 73.18 │ 97 │ 4990.721649484536 │ 5025.0 │ │ 19.09 │ 73.2 │ 97 │ 5232.216494845361 │ 5175.0 │ │ 19.09 │ 73.14 │ 96 │ 4284.635416666667 │ 4275.0 │ │ 19.09 │ 72.91 │ 94 │ 819.4148936170212 │ 862.5 │ │ 18.58 │ 74.01 │ 92 │ 3408.4239130434785 │ 3400.0 │ │ 19.09 │ 73.0 │ 92 │ 2288.586956521739 │ 2300.0 │ ... ... ```  A clear strip can be seen with the aircrafts descending into the runway strip. Also the dip is more pronounced in the Navi Mumbai / Airoli / Ghansoli / Thane-side approach corridor. I also looked for a query on the Ghatkopar region, and it was amazing to see that it was so near. Around `19.08, 72.94` is Ghatkopar, and aircrafts have an approximate average of 800 feets. 90% of them are below 1,100 feet, that is crazy low. And that clearly explains why the strip straight to the East-West of the airport line (which some part of Ghatkopar falls into) doesn't have any tall towers. ``` ┌────────────────┬───────────────────┬───────────────┬────────────┬────────────────────┐ │ descent_points │ avg_alt_ft │ median_alt_ft │ p90_alt_ft │ p95_alt_ft │ │ int64 │ double │ double │ double │ double │ ├────────────────┼───────────────────┼───────────────┼────────────┼────────────────────┤ │ 395 │ 859.7544303797469 │ 775.0 │ 1175.0 │ 1332.4999999999989 │ └────────────────┴───────────────────┴───────────────┴────────────┴────────────────────┘ ``` Another good metric to verify the dataset is proving right. ### Where Takeoffs Start Takeoffs are the same idea in reverse. The question is not where the plane ends up in the sky. The question is where it starts climbing away from the airport. So instead of looking for a finished takeoff, I look for the first point in each run where altitude clearly begins to rise. That gives me the start of the departure path. Then I group those start points by location. If the same area keeps lighting up, that tells me where departures are really getting going. That is why the takeoff map is tighter and closer to the airport than the landing map. It is not trying to show the whole flight. It is only trying to show the first step out of the airport, which is the part that makes the pattern visible. The sharp climb starts very close to the airport itself, and that is what the takeoff pattern shows clearly. The aircrafts begin to gain altitude almost immediately after leaving the runway, so the takeoff strip stays tight around the airport side instead of spreading out far into the city. - `19.09,72.81` and `19.09,72.82` sit just west of the airport and catch the early climb corridor right after liftoff. This is the sea side, beyond the Juhu aerodrome, perfect. `19.09,72.78` to `19.09,72.80` are a little farther out, but they still belong to the same western departure corridor, showing that the climb pattern stays compact near the field. - `19.09,72.84`, `19.09,72.85`, `19.09,72.86`, and 19.09,72.87 are even closer to the airport edge, so they sit in the immediate departure strip where the climb becomes visible first. - ``` ┌────────────────────┬───────────────────┬────────────────┬────────────────────┬───────────────┐ │ lat_bucket │ lon_bucket │ takeoff_points │ avg_alt_ft │ median_alt_ft │ │ double │ double │ int64 │ double │ double │ ├────────────────────┼───────────────────┼────────────────┼────────────────────┼───────────────┤ │ 19.09 │ 72.81 │ 1473 │ 2092.6171079429737 │ 2050.0 │ │ 19.09 │ 72.82000000000001 │ 1446 │ 1889.3672199170123 │ 1825.0 │ │ 19.09 │ 72.84 │ 1374 │ 1094.5778748180494 │ 1075.0 │ │ 19.09 │ 72.85000000000001 │ 1329 │ 764.8796087283672 │ 725.0 │ │ 19.09 │ 72.83 │ 1208 │ 1518.4395695364237 │ 1450.0 │ │ 19.09 │ 72.87 │ 1079 │ 465.8016682113068 │ 200.0 │ │ 19.09 │ 72.8 │ 994 │ 2233.6770623742455 │ 2150.0 │ │ 19.09 │ 72.86 │ 848 │ 698.1816037735849 │ 400.0 │ │ 19.09 │ 72.79 │ 538 │ 2579.460966542751 │ 2525.0 │ │ 19.09 │ 72.78 │ 486 │ 2790.792181069959 │ 2750.0 │ │ 19.09 │ 72.77 │ 418 │ 3224.5215311004786 │ 3000.0 │ │ 19.09 │ 72.76 │ 331 │ 3665.7099697885196 │ 3475.0 │ │ 19.09 │ 72.75 │ 234 │ 3924.5726495726494 │ 3662.5 │ │ 19.09 │ 72.88 │ 206 │ 976.2135922330098 │ 200.0 │ │ 19.09 │ 72.74 │ 137 │ 4216.605839416058 │ 3875.0 │ │ 19.09 │ 72.73 │ 129 │ 4780.232558139535 │ 4925.0 │ │ 19.09 │ 72.72 │ 68 │ 4781.985294117647 │ 4837.5 │ │ 19.18 │ 72.74 │ 62 │ 7492.741935483871 │ 7325.0 │ │ 19.19 │ 72.75 │ 62 │ 7558.870967741936 │ 7525.0 │ │ 19.080000000000002 │ 72.73 │ 54 │ 5360.185185185185 │ 5162.5 │ ... ... ```  There is no major analysis here, since it has plain ocean in front of it, we can't really track the location thereafter. We can just use the direction that we'll check separately. ### Frequency of landings and takeoffs I wanted the spacing between landings on their own, and takeoffs on their own. So I grouped the airport events by type, sorted them in time order, and measured the gap between one landing and the next landing, and one takeoff and the next takeoff. The median is the number to trust most here, because a few long gaps pull the average up. The process was simple: - Classify events as landing or takeoff - Sort by time inside each sampled day - Compare each event to the next event of the same type - Measure the minutes between them  In this sample, landings are spaced about 2.58 minutes apart at the median, and takeoffs about 3.0 minutes apart. What can we observe from this? - Landings are spaced a bit more tightly than takeoffs - Takeoffs have a slightly longer pause between consecutive events - The median is the number to trust most here, because a few long gaps pull the average up This is the answer I have been chasing all along and it truly feels great to see it that close. As children we had observed the landing frequency as around 2 minute 30 seconds, and it couldn't be more accurate than this. But the great thing about the BOM/VABB Airport is that it still holds true even after a decade. It truly is the single runway busiest airport. ### Landing-to-Takeoff Gaps I want to measure the time between a flight landed and another took-off (not the same aircraft), any flight, like just after a flight landed, how much time it took for another to takeoff. I think it has to be in the range of 1-2 minutes. That's the status BOM/VABB Airport has to be the busiest single run way airport. Let's check and validate it with data. To answer that, I look at the airport events in time order and take each landing as a starting point. Then I find the next takeoff that follows it and measure the time gap between the two. If that gap is small, it means the airport is keeping the movement going very quickly. If it is large, it means there was a longer pause before the next departure. The interesting part is not the average alone, but the shape of the gaps across the sample. In this sample, that gap is very short most of the time. The median is 1.5 minutes, and the average is about 2.1 minutes, across 5,297 landing-to-takeoff intervals. Most of the bars are packed into the first few minutes, which tells me the airport is turning movements over very quickly. The long tail is there too, but it is small. So the real story is not "there are a few huge delays" it is that the typical landing is followed by a takeoff almost immediately.  And another question with intuition validated with data. If you think about it for a while, the median is 1.5 minutes, if you do the math (60/1.5 = 40 movements / hour). That alligns well with the hourly average. And it even validates the historical peak of Mumbai of 51 movements with 1,036 daily movements, that number is not far. It is that the airport's fame comes from how little space there is between operations. The chart makes that visible: most of the time, the next takeoff follows a landing almost immediately, and that is exactly the kind of traffic that built BOM's reputation. ### Weekday Pattern This is a bit interesting question, since I honestly had no clue about the answer. I thought every day would have the same pattern. I was wrong. To get this, it was straight forward. I took the airport events I had already classified as landings and takeoffs, grouped them by weekday in Mumbai time, and counted how many movements landed in each bucket. That gives a simple answer to a simple question: which day of the week gets the most airport activity? This question is asking about the breakdown of the flights based on days of the week. ``` ┌─────────────┬──────────────┬──────────┬──────────┬───────────┐ │ weekday_num │ weekday_name │ landings │ takeoffs │ movements │ │ int64 │ varchar │ int128 │ int128 │ int64 │ ├─────────────┼──────────────┼──────────┼──────────┼───────────┤ │ 0 │ Sunday │ 1444 │ 1239 │ 2683 │ │ 1 │ Monday │ 1383 │ 942 │ 2325 │ │ 2 │ Tuesday │ 1695 │ 1337 │ 3032 │ │ 3 │ Wednesday │ 1748 │ 1053 │ 2801 │ │ 4 │ Thursday │ 1805 │ 1236 │ 3041 │ │ 5 │ Friday │ 1859 │ 1323 │ 3182 │ │ 6 │ Saturday │ 2789 │ 2070 │ 4859 │ └─────────────┴──────────────┴──────────┴──────────┴───────────┘ ```  The result is pretty clear in the sample. Saturday stands out as the busiest day by a wide margin, with 4,859 movements, followed by Friday at 3,182 and Thursday at 3,041. Monday is the quietest at 2,325. > Landings are higher than takeoffs on every weekday, but the weekend spike is the real signal here. So the sample does not look evenly spread across the week, it has a strong Saturday peak and a weaker Monday trough. I then looked into the dataset and tried to find the count of each day in the sample, since i had around ~70 days of data (1 day * 12 months * 6 years) - Saturday 12 - Sunday 11 - Wednesday 11 - Friday 11 - Tuesday 10 - Thursday 10 - Monday 8 And that can explain why Saturday is the busiest day. The sample is not evenly distributed, but it is more or less evenly distributed across the calendar. ### Hotspots / Direction The question here is when the aircrafts starts landing or taking off, what direction are they actually pointing? That's quite obvious right? But I want to identify a deeper pattern here, from where are they coming from and where are they heading towards. That matters because it tells you whether Mumbai's traffic is spread randomly across the compass or funneled through one stable corridor. If the airport is using the same flow again and again, the headings should cluster into a small set of directions instead of being evenly scattered. To get this, we can: - take each inferred flight run - classify as a landing or takeoff - capture the heading at the relevant point in the run. For landings, it uses the heading at the end of the run. For takeoffs, it uses the heading at the start.   Those headings are then grouped into direction sectors and counted, which is what produces the direction chart.   The result is very strong and very repetitive. Most landings cluster around 270°, with a smaller 90° cluster, and takeoffs are also dominated by 270°. In the heading data, the top landing sector is 270° with 14,316 points, and the top takeoff sector is also 270° with 10,848 points. That tells me the airport is not behaving like a random cloud of movement. It is using the same directional axis again and again, which is exactly why the hotspot map looks like a corridor instead of a scatterplot. ### Time in Airspace The question here is are the aircrafts just passing through, or do they stay around Mumbai for a while? The idea is simple, if landings take longer than takeoffs, that usually means approach and descent are keeping the aircraft in the airspace longer. One thing I wanted to know was how long aircraft actually stay around Mumbai once they enter the airport flow.  So I looked at the duration of each inferred run and compared landings, takeoffs, and the full set together. The pattern is pretty clear: landings stay in the airspace longer than takeoffs. The median landing run is 34.5 minutes, while the median takeoff run is 25.1 minutes. Across all airport events, the median is 27.3 minutes. That tells me the traffic is not just passing through fast. A lot of it is lingering long enough for the airspace to feel full, especially on the landing side. The interesting part here is not just that landings last longer than takeoffs. It is what that tells me about the way Mumbai moves aircraft through its airspace. A landing is not a single movement. It is a long descent, even slower, and then the final stretch into the runway, so it naturally stays visible in the data for longer. Takeoffs are different. Once an aircraft leaves the runway and starts climbing out, it clears the airport zone much faster. That is why the landing median is 34.5 minutes, while takeoffs sit closer to 25.1 minutes. This does not mean the aircraft is simply flying slower in the sky. It means the arrival side of the airport keeps aircraft in the flow for longer, which is exactly what you would expect at a busy airport like Mumbai. It also hints at sequencing and approach management: arrivals are being held in the pattern of the airport longer than departures, which is the kind of behavior that makes the airspace feel crowded even before you count the total number of flights. So the real takeaway is this: Arrivals spend more time inside the airport's operating envelope, while departures clear out faster. That difference is part of the airport's style, and it helps explain why the airspace feels full even when the hourly flow looks ordinary on paper. ### Aircraft Turnaround Time > After this aircraft lands, how long does it stay on the ground before it takes off again? This is a good question, because it gives a deeper look at the activity on the airport. It doesn't necessarily give much but still a valid metric to observe. So I matched each landing with the next takeoff for the same aircraft on the same day. That gives a proper aircraft-level ground-time measure, instead of a flight-level handoff.  What I observed: The turnaround time is steady, but not extremely fast. The median ground time is 96.5 minutes, which is about 1 hour 37 minutes. The middle half of the aircraft stays between 77.5 and 130.5 minutes, and the 90th percentile goes up to 185.3 minutes. Most of the pairs sit in the 60-180 minute range, which suggests a working fleet that is being reused at a regular pace rather than left idle for long stretches. That also hints at the kind of airport Mumbai is. This looks much more like a high-frequency rotation airport than a place where aircraft disappear for half a day between uses. A lot of the traffic is probably made up of domestic and short-to-medium-haul operations, where the same aircraft can land, sit for a bit, and then go back out again within a couple of hours. So the takeaway is not just "there is a gap" It is that Mumbai keeps the aircraft moving in a fairly tight loop, which fits the airport's reputation for being heavily used and constantly in motion. ### International vs Domestic Flights I also wanted to see how much of Mumbai's traffic is Indian-registered versus foreign-registered. I could not see the route directly in the ADS-B feed, so I used aircraft registration as a proxy: `VT-` for Indian-registered aircraft and everything else as foreign-registered. That is not the same thing as domestic vs international routes, but it gives a useful high-level split. ``` ┌────────────────────────┬───────────┐ │ bucket │ movements │ │ varchar │ int64 │ ├────────────────────────┼───────────┤ │ Unknown │ 3399 │ │ Indian-registered (VT) │ 110492 │ │ Foreign-registered │ 37589 │ └────────────────────────┴───────────┘ ```  The result is clear: about `72.9%` of movements are Indian-registered, while `24.8%` are foreign-registered. So the airport is still dominated by Indian traffic by count, but foreign aircraft are not a side note, or even a minority. They are a substantial and regular part of the flow. ## What Surprised Me What surprised me most was not that Mumbai is busy. I already expected that. What surprised me was how tight and consistent the whole system is. The airport does not just have a lot of movement, it has a deadly consistency to it. A landing is followed by a takeoff almost immediately, the same aircraft comes back into service in about an hour and a half, and the busiest hours are not random bursts but repeatable peaks that keep showing up in the same parts of the day. The other thing that stood out was how directional the traffic is. Once I started looking at landings and takeoffs on the map, the pattern stopped looking like scattered noise and started looking like a corridor. The airport is not using the full sky evenly. It keeps leaning on the same approach and departure axis, which makes the movement feel much more concentrated than a simple flight count would suggest. I mean that is no brainer, since it can disrupt the residential living, the noise and the unpredictable zoning of planes is not airports should be doing. And Mumbai does it very neatly. I also did not expect the weekday pattern to be so uneven. I assumed the traffic would be roughly similar across the week, but Saturday clearly pulled ahead in the sample. That was a good reminder that airports are not just shaped by the clock, they are also shaped by the calendar. Even with a dataset this limited, the weekly pattern still shows through. There is the big myth which gets debunked here, that is "Mumbai Airport is busiest in the late night". Opposite is true here. The airport is busiest in the morning and evening, and quietest in the late night. The airport is not empty for long. Though there might be international flights in the late night but they don't land and takeoff with the frequency as close as Indigo does, so that Mumbaikars can sleep finally. Another small but satisfying surprise was that the numbers matched the memory. As a child, I had the feeling that Mumbai's sky was never empty for long. The data backed that up. It felt satisfying to see the numbers in front of me for the frequency of landing, the takeoff to landing gaps, the dominance of airline, and mapping them to intuition and memories. The frequency is consistent like crazy, the turnaround period really tight, people do have a favorite preference of Indigo, everything just feels calm but its really quick and consistent. The airport really does stand up to its name "Single Runway Busiest Airport". ## Conclusion Well! That was a lot I admit. But it was fun. Atleast to me (I had said, it was a post for me!). I hope you got a little smile out of it if not just felt good reading about how Mumbai's Airport Operate. This is still ADS-B data, so it is a proxy, not an official operational log. But for the question I started with, it is enough. The feeling I had as a kid was real, Mumbai's sky was never empty for long. The frequency of landing is truly at 2 minute 30 second roughly speaking, which just makes me feel joyous, the airport turnaround time is quite a sharp and tight one, making it really a Mumbai-like busy runway. But that is just part 1 buddy! Haven't nerded about the complications of ADS-B data and my struggles of it, fighting with the limited disk space to compress the data and juggernauting queries to get results. That is the next part. Check out the full case study here -> [Flight Observatory - Mumbai Airspace Case Study](https://dev.meetgor.com/flight-observatory/case-study/mumbai-airport/) https://www.meetgor.com/posts/2025-review I had focused on tech on this reviews. I would keep it same this year, I might write a separate non-tech, a human yearly review on my separate blog. Gist I move Wed, 31 Dec 2025 00:00:00 UTC I had focused on tech on this reviews. I would keep it same this year, I might write a separate non-tech, a human yearly review on my separate blog. ## Gist I moved from a software developer to a reliable and responsible developer. I took ownership and sat on problems more than the level of my comfort zone. It made me a better developer, it helped me sympathize the people around me, and connect more. These moments might create life long relations, but from the macro perspective they looked frustrating and annoying at times. Here I am still finding peace and standing and walking through what has not been a smooth journey both at work and life. I was thinking work and life could mean same thing once I got out of college, but those kept diverging and I had to keep both my feet at par with each one's pace. I hit you with quotes and I will again > "Its easy to judge anyone, its harder to be someone" I have read a lot of books, and one quote just hits me is this, I will explain this in my non-tech review post. Its quite easy to point out flaws and judge the other person from outside, but the same person if you were in his or her shoes, its quite hard to the otherwise. It's easy to judge a developer, but people slowly realized vibe coding hits a halt at some point, you are not replacing a developer, you are trying to replace someone who cares deeply about the problem and not someone who just can write code. Vibe coding rose and fall with the same speed. ## Key Achievements 1. Got promoted as a Software Developer from Junior (completed 2 years at Docsumo 8 month internship + 16 month full time) 2. Wrote 52 newsletter editions consistently reflecting on weekly learning I didn't finish any goals I wrote in 2024 review, - Write blogs for FreeCodeCamp (at least 3) (none written) - Learn Kubernetes in detail (not yet) - Write 25 articles on 100-days-of-Golang (wrote 3) - Learn Rust or Zig (Nope) - Get 2K subscribers on YouTube (Nope, not even 100) I want to straight honest here, things change rapidly in 2025, I barely had things in control. I was consistent, I had time, yes! But I had to pivot to other goals to keep my self sane. Not clearing 5 goals doesn't mean I cleared none I did - Understand AI Agents before the hype (early January) - Learn in depth about SQL and SQLite with a learning log - Complete Advent of SQL (15 blog posts) Also some broader or abstract goals like - Adapt LLMs in the development process - Reflecting more than consuming - Understanding one technology in depth (sqlite) - Complete things rather than shove it off after a while (project graveyard, or irresponsible employee) Let's break it down to 3 things 1. Work 2. Side hustles 3. Learning ## Work I work at Docsumo, so I work with documents. The field of intelligent document processing is taken by the storm by LLMs. Prior to LLMs, there were ML and rule based systems to extract data. Now we have LLMs, we not only can extract structured data, but can have a conversation with the document. It has changed the way document are treated. We decided to go full on LLMs and use the latest model for extracting data. It was tough, there were issues, wired behaviors and of course hallucinations. Most of them were related to prompt but also more on how we call the LLM, through plain api, with structured output, or as an agent. We started with the first simple api call, sat on it for quote some months, but external factors forced me to use other tools to find the extraction more accurate. I started experimenting with LLMs more than ever from the 2nd quarter of 2025. LLMs were released every week with minor improvements or major shift in tool calling. Everything had to go through the LLM, write and review code with Coding agents, use LLM to extract data, debug with LLMs. There was a tremendous shift in how we develop software and I was blessed and lucky to be in a company that helps developers stay up to date with the cutting edge tools like Cursor, Windsurf, Code Rabbit, Gemini, OpenAI playground and what not. I was given these tools, as a new toolkit. No one actually knew how to use them, they knew the general sense of intuition, but it was like a alienated tool. Everyone gave their own shot at using it, and most them hated it at first, but then they understood the pattern, changed the way I and they used it and almost in no time it became the default editor and tool that we use now. I never new change in software was that rapid! Maybe this is a special time, but every time we say that. I honestly was reluctant to use AI tools as daily driver at first, but saw the speed of others and it bogged me down that I was missing out on something. And that was code-review and hands off understanding of the code. It was very hard for me to understand and debug the code that I haven't written, I just accepted it without understanding, because it feels daunting to review code. Especially when you haven't written and reviewed a lot of code, human code. Its almost like reviewing the classical books (from Shakespere and Tolstoy) after reading Harry Potter. You don't have the vocabulary to understand those books, you can't review things that you don't know the patterns and the nuts and bolts of. It is possible but its quite tedious and overwhelming. "Overwhelming" this is a underrated term for what I want to describe how I felt working throughout the year. Coding was supposed to be hands on, but when exposed with a magical bot that can write and churn hundreds and thousands of changes in few minutes is just not processable from my tiny brain. I failed badly at using these tools, with skeptism. But then slowly over couple of months, with the right task coming at hand, and trying to throw anything at it, I understood. What can work and what can't work given a task to an LLM. I realized the hard way, that LLMs are good for throw-away code. The code that you can just use it for producing something that you don't care about how it did it. Like log analysis, scripts for downloading data, filtering files based on some names, analysing and combining bunch of CSVs, I don't care if you use proper variables and correct abstractions to do those things. These are the things where I never second guessed myself to throw at LLM and take whatever it spat out with a grain of salt and modified it till it made sense. I am still finding it hard to write everything of that sort to LLM, as nothing is left to do! I feel bored! Coding was supposed to be fun, a bit time consuming but a good exercise to the brain. But people have started preferring velocity over craft. I am not against it, but sometimes we get too carried away with human ambition. I hope we slow down in 2026, though I am skeptical about it, it might just get worse. We need time to understand and explore what can be do with the current set of tools and technologies that these AI labs and all of us have created and release just in the last year. There is absurd amount of things that we need to build with them, in fact I think we can stay as is without new models or breakthrough for decades with just these tools. But Damn! We are not stopping here, are we? I think in 2026, I need to build more and get a taste and expertise in systems. I am an engineer and writing code is already obsolete. The only thing that can stand out is understanding how things can fit and change in existing or new system software. Just technical metrics, I don't care but gives me a sense of wonder of wow! that was quite a lot! - 336 commits on core services that I maintain - 101 PRs closed - Favorite model? GPT 5 (I am surprised cursor) - 708 M tokens burnt (geese) ## Side Hustle Side projects of course! I had 100+ ideas but never picked up anyone of them seriously. What is wrong with me? Is it due to AI, no. It was same as before AI too. AI just made it easier to reach that point. With LLMs and these CLI coding agents, I can scaffold projects in seconds than in minutes. I initially thought it was cool, so that I can actually build the core thing, once the boilerplate code is created by LLM. But boy! I was wrong. It creates a layer of understanding that I then have to dread through, then with the half mine and half LLM baked system, I have to steer myself or the LLM in completing the project. And that's the bottleneck I am not able to solve it yet. I am writing this and realizing the problem now. Reflection is key in understanding problems. Writing is the easiest way to it. I don't have much to say, I tried building CLIs, TUIs, Web apps, all failed due to my non-thinking of the parts, and was too lazy to write myself or LLM was not able to understand me, my prompting skills have gotten better over time, but I am bad at expressing things in plain english quickly, I need time, but the excitement of seeing it dampens it. Side project graveyard is ever expanding, and we need something to cure it! LLMs have made it easier to expand, but also they have to power to complete them, the skill is the gap that is preventing me from achieving just that. 2026, Fingers crossed. I can solve at least one problem that people can relate to and love. ## Learning This I think is the least interesting thing for people right now, LLMs have made it look obsolete to learn what's new in SQLite 3.51, or how to use f and t strings in python 3.14. But I think it had given us ample reasons to learn anything. Why? Developer are paid to fix problems, not write code. If you know certain things, you always and always will have advantage over LLMs. You can be smart by leveraging LLMs to learn, in the thinking mode, it can spit out legit facts, but do the opposite thing of what it intended to do, you as a developer can see and do things better than LLMs. This is still a problem in most of the overthinking LLMs. They are getting better, but a human thinking will always out-power an LLM with fake thinking. I decided to learn SQLite in 2025, and will continue to learn in 2026, its something that cannot be completed. I might also start learning Kubernetes the same way, though I wanted to learn it since 2 years but never started. People will soon realize that learning is something that they can't outsource to an LLM, nor the thinking. Because humans at core are curious creatures, if you think you can live without learning anything, than you are not a software developer. Simple. Learning is an lifelong process, nothing can stop it, it could be never completed. There will always be something to learn. For instance, have you ever wondered how LLMs can read files? Like PDFs and words or images, since the data is not a clean text, how does it parse it? Well that is a rabbit hole for learning which could benefit you over just using LLM to churn and do a trial and error for correct result. If you have a tool, you need to understand it to the finest to hone and master it, for no such tool can be mastered. > Learning to learn is going to be more important than building in the future That's just my opinion! Who knows, human just take AI slop and feed it to the next AI until they have something they like and never learn! That would be the worst of time for humans like me to live. - 52 Newsletters published at every Saturday midnight IST - 53 SQLogs written - 3 in-depth posts about Golang - 458 commits - 209 day reading streak on daily.dev ## Lessons Learnt 1. Never outsource thinking 2. Be with people and speak out 3. Habits are actually easier to build, the temptation to skip it for today is what gets better of me 4. Developers were never for writing code, they were for caring about the problem the deepest way 5. I can learn anything, the only thing in my way is my mindset and distractions Keeping mind peaceful from outside though a lot is burning in the house, is something I juggled with. ## Off to 2026 There are surley certain things I want to do explicitly in 2026 like 1. Understanding databases on a deeper level 2. Understanding LLMs on a deeper level 3. Building tools with LLMs for fun 4. Writing on my blog with less friction to publish thoughts, journals and devlogs or TILs 5. Building educative material related to databases and SQL Yes, these are very specific, but I don't like lame and broader goals like "Be a better developer", "Understand systems", and what not. I want to dive deep, I want to replace scrolling with action. 2025 was wild, it was slow yet models kept falling like dog water! I don't expect for LLMs to go in 2026, that is the last thing that could happen, but I can see the thing starting to build around them, tooling for understanding them, helping with the smallest of things human care for, its already in all the things we use, yet people don't use it as reliable tools. The ones they do, they might soon realize what kind of fire they are playing with. Anyways! I hope everyone a Happy New year and may 2026 be kinder, more fulfilling to you. Keep Thinking! https://www.meetgor.com/posts/golang-web-delete-method Introduction We have explored GET, POST, PUT, and PATCH methods in our previous entries for this series. It is the final entry for all the HTTP methods which is Sun, 12 Jan 2025 00:00:00 UTC ## Introduction We have explored GET, POST, PUT, and PATCH methods in our previous entries for this series. It is the final entry for all the HTTP methods which is going to be the `DELETE` method. In this entry of the series, we will take a view on how to construct and request an HTTP DELETE Method to an API Endpoint. The DELETE Method is quite simple. You just provide the URI of the resource. Most often, the request body is not needed. That request will simply delete the request entity from the server (in the database or wherever the resource is stored). Let’s understand it in a more detailed way. ## What is a DELETE Method The DELETE method requests that the server remove the association between the target resource and its URI (Uniform Resource Identifier). This doesn't necessarily mean the underlying data is physically deleted; it means the resource is no longer accessible through that specific URL. DELETE can also be used to remove relationships between resources, effectively "delinking" them. According to the RFC: > The DELETE method requests that the origin server remove the association between the target resource and its current functionality. Examples: * **Social Media** (Deleting a Tweet): When you delete a tweet, you're sending a DELETE request to the server. This removes the tweet from your timeline and makes it inaccessible via its URL. While the data might be archived or retained for a period, the key action is removing the public association between the tweet and its online presence. This is closer to a true deletion than the cart example. * **E-Commerce** (Removing an Item from a Cart): When you remove an item from your online shopping cart, you're sending a request (often a DELETE) to remove the item from your cart. The actual product remains available in the store's inventory. This is a clear example of delinking. You're deleting the link between your cart and the product, not the product itself. Let’s start constructing a simple DELETE Request in Golang. ## A Simple DELETE Request We don’t have a specific method for `DELETE` in `net/http` as we have for `GET` and `POST`, so we need to create a request and use a client to send the request. ### Constructing the URL We would need to define the endpoint that we are hitting. We can directly use the API URL or construct the API URL on the fly, depending on the ID and dynamic parameters. DELETE requests usually delete a particular entity. We would generally have some form of identifier for that entity/object on the database, etc. So, in this case, it is the user's ID, so we can pass the post to the URL. ```go // define URL to hit the API apiURL := "https://reqres.in/api/users/4" // OR baseURL := "https://reqres.in" userID := 4 apiURL := fmt.Sprintf("%s/api/users/%d", baseURL, userID) ``` We can either directly define the URL or dynamically construct the URL, that is quite straightforward. The latter one is the one we usually use and design. The DELETE Request doesn’t usually require a request body, however, if your server requires some specifications, you can construct the body as we did with the previous examples in POST, PUT, or PATCH method requests. ### Constructing and sending the DELETE Request We can simply construct the request by specifying the http.MethodDelete as the request method, the URL to hit, and a body(optional) just like a `GET` request. Once we have the request, we can create the default client and send the request. ```go // create a DELETE request req, err := http.NewRequest(http.MethodDelete, apiURL, nil) if err != nil { log.Fatal(err) } // construct the default http client and send the request client := &http.Client{} resp, err := client.Do(req) if err != nil { log.Fatal(err) } ``` This is the normal code used for constructing an HTTP request in Golang, we create a request using the NewRequest function that takes in the method type, the URL to send the request, and the body if any. Then we need to create a http.Client for sending the request, we usually create a client with default values and send the request using the Do method on the created client using the request that we constructed earlier. ### Fetching the Response Once the request is sent, we can fetch the response and read the body as bytes, and check the status if that succeeded or failed. ```go fmt.Println("Response Status:", resp.Status) respBody, err := io.ReadAll(resp.Body) if err != nil { log.Fatal(err) } fmt.Println("Response Body:", string(respBody)) ``` We can grab the Status field for checking the status code and message for the request. Usually, the body would be empty since there is no resource we are expecting after deletion of the object. However, if the server is implemented in a way to return the deleted object, you can read the bytes of the body and unmarshal it to the desired struct. So, that is the entire code to create a simple Delete request with Go, simply construct the URL with the identifier for the resource to be deleted, create the request, and send the request, and if the status code is 204 (usually) then we can assume it succeeded. ```go package main import ( "fmt" "io" "net/http" ) func main() { baseURL := "https://reqres.in" userID := 2 apiURL := fmt.Sprintf("%s/api/users/%d", baseURL, userID) req, err := http.NewRequest(http.MethodDelete, apiURL, nil) if err != nil { fmt.Println(err) return } client := &http.Client{} resp, err := client.Do(req) if err != nil { fmt.Println(err) return } fmt.Println("Response Status:", resp.Status) respBody, err := io.ReadAll(resp.Body) if err != nil { fmt.Println(err) return } fmt.Println("Response Body:", string(respBody)) } ``` ## Facts about the DELETE Method * DELETE Method is idempotent: Similar requests will result in identical behavior or response since once the resource is deleted, the resource won’t exist and hence the behavior would not hinder any other parts. * DELETE Method is not safe: The operation is quite dangerous as it could literally remove a resource from a database/storage. Hence it is called not safe as it is making changes on the server. I have also included some more examples of DELETE requests [**here**](https://github.com/Mr-Destructive/100-days-of-golang/blob/main/web/methods/delete/). That's it from the 36th part of the series, all the source code for the examples are linked in the GitHub on the [**100 days of Golang**](https://github.com/Mr-Destructive/100-days-of-golang/tree/main/web/methods/delete/) repository. [**100-days-of-golang**](https://github.com/Mr-Destructive/100-days-of-golang) ## Conclusion That would be it from the DELETE Method in Golang. We can use this method just like a normal `GET` request however a bit more carefully. Hope you found this article, helpful, leave some feedback or any suggestions if you have any. Thank you for reading. Happy Coding :) https://www.meetgor.com/posts/golang-web-patch-method Introduction In previous sections of this series, we've covered the GET, POST, and PUT methods. Now, we will explore the PATCH method, which differs from the ot Sat, 11 Jan 2025 00:00:00 UTC ## Introduction In previous sections of this series, we've covered the GET, POST, and PUT methods. Now, we will explore the PATCH method, which differs from the others in several key ways. The PATCH method is somewhat more flexible and depends on how the server or API you're working with is designed. In this section, we'll focus on understanding what the PATCH method is and how to use it. While we will dive deeper into building and structuring a full CRUD API later in the series, the focus here will be on the what and why of the PATCH method, not the how. ## What is the PATCH Method? The PATCH method is often compared to the PUT method, but with one important distinction: PATCH is used to perform partial updates on a resource. Unlike PUT, which typically requires you to send the entire resource to update it, PATCH allows you to send only the fields that need to be updated. This makes it a more efficient option when updating a subset of a resource. In a PATCH request, the body usually contains instructions in a format like JSON, which specifies the fields to update. These instructions define the changes to be applied to the resource. For example, you may only want to change one field of a user's profile, such as their email address, while leaving the rest of the data untouched. ## PATCH vs. PUT Key Differences While both PATCH and PUT are used to modify resources, there are significant differences in their behavior: * PUT replaces the entire resource. When you send a PUT request, you must include the full representation of the resource, even if you're only changing a small part of it. * PATCH, on the other hand, is for partial updates. You only need to include the fields that are changing, not the entire resource. If the update involves more fields than just the ones you're changing, PUT may be the better choice. However, the scope of this article is to focus solely on the PATCH method. ## How Does a PATCH Request Work? In the simplest terms, a PATCH request allows you to perform a partial update on a resource. It is similar to a PUT request, but more specific in how it updates the resource. According to the HTTP specification, an ideal PATCH request should: * Accept a "patch document" in the request body, which contains the list of operations to apply (e.g., "replace", "add", "remove"). * Apply these updates to the target resource. * If the update cannot be applied correctly, the operation should fail without applying any of the changes. This ensures that no partial or inconsistent updates are left behind For example, if you're updating a user's email address and something goes wrong in the middle of the operation, the PATCH request should ensure that the email isn't updated partially. If there’s an error, none of the updates should be applied, ensuring data consistency. Also, the patch method is not idempotent, meaning that if you send the same input/request, it need not necessarily return the same output. Because we are not sending the actual original entity, we are sending the partial data fields that need to be updated, so it might update the original entity on subsequent requests since there is no original request sent in the request; it only identifies the resource from the URI and fields to update in the request body. Now, let’s sum up the patch request in a few words * Updates specific fields mentioned in the patch document * Can be partial (only the fields that need to be updated are sent, unlike PUT, which typically replaces the entire resource) * Not necessarily idempotent (depends on the implementation) * Not Safe (since resources will be updated on the server side) ## Basic PATCH request Let’s start with the basic PATCH request that we can create in Golang. The \[net/http\]() package will be used to construct the request, and will be using `encoding/json` and some other utilities for string and byte parsing. So, first we will construct a HTTP request using the \[http.NewRequest\]() with the parameters like the http method to use, the URL to hit, and the request body if any. We will then need to send the json body which would consist of the fields to be updated. ### Defining the API/Server Endpoint URL We would need to define the endpoint that we are hitting, we can directly use the API URL or we can construct the API URL on the fly depending on the id, and parameter that will be dynamic. As PATCH request, usually modify a particular entity, we would generally have some form of identifier for that entity/object on the database, etc. So in this case, it is `id` of the post, so, we can pass the post in the URL. ```go // define URL to hit the API apiURL := "https://jsonplaceholder.typicode.com/posts/4" // OR // baseURL := "https://jsonplaceholder.typicode.com" // postId := 4 // postURL := fmt.Sprintf("%s/posts/%d", baseURL, postId) ``` We can either directly define the URL or dynamically construct the URL, that is quite straightforward. The later one is the one we usually use and design. ### Constructing the JSON Body This section is a little dependent on the context as you might have a direct json string that you can directly pass to the API or you might have a golang object that you need to Marshal in order to convert that object into string/bytes. 1. Direct JSON String So, there is nothing to do here, since the object is already in the form of a json string. ```go reqBody := `{"body": "new body"}` ``` However, if you have certain fields that you need to exclude or omit, you have to construct a struct and then marshal it 2. Marshalling (converting object into bytes/string) We need to convert the Golang native object into some form of a json string or bytes that could be sent over the network. That process is called [marshalling](https://en.wikipedia.org/wiki/Marshalling_\(computer_science\)) or serialisation. ```go type Post struct { ID int `json:"id,omitempty"` Title string `json:"title,omitempty"` Body string `json:"body,omitempty"` UserId int `json:"userId,omitempty"` } userObj := Post{ Body: "New Body", } var reqBody []byte reqBody, err := json.Marshal(userObj) if err != nil { log.Fatal(err) } log.Println("New body:", string(reqBody)) // New body: {"body":"New Body"} ``` In the above snippet, we have defined a `Post` struct with the fields like `ID`, `Title`, `Body`, `UserID` , and those have `omitempty` tag along with the json fields that we want to marshal into. The omitempty will omit or ignore the fields if they are empty or not present in the object/instance of this structure. So in the example, `userObj` is the instance of the `Post` struct and it only has the `Body` populated, so the reqBody will only contain one field `body` in the json representation. The \[json.Marshal\]() is the function that we use to convert the object/instance into a byte form. This `reqBody` will serve as the request body for the request that will be a `PATCH` method to the mentioned endpoint / API URL. ### Constructing the HTTP PATCH Request Now, we have the parts that we need to construct the request, we can combine the parts and hit the endpoint. However, it is a bit different compared to `GET` and `POST` request that we do in Golang, the HTTP package has built in methods for the `GET` and `POST` methods, however for methods like `PUT`, `PATCH`, `DELETE` and others, we need to construct a [Request](https://pkg.go.dev/net/http#Request) object and then send that request. ```go req, err := http.NewRequest("PATCH", postURL, strings.NewReader(reqBody)) if err != nil { log.Fatal(err) } req.Header.Set("Content-Type", "application/json") // in case of wired utf-8 characters appear in the body //req.Header.Set("Content-Type", "application/json; charset=utf-8") ``` To do that, we call the [NewRequest](https://pkg.go.dev/net/http#NewRequest) method with the parameters like the HTTP method, the URL, and the request Body all of which we have at the moment. * The method is `PATCH` * The URL is `postURL` * The body is `strings.NewReader(reqBody)` as we need a `io.Reader` object instead of string or byte slice So, once we have that, we would also set the `Header` with the field of `Content-Type` and the value as `application/json` since the request body has json representation of the patch document that will be sent. ### Sending the Request Once, the `req` object is created, we also need a [Client](https://pkg.go.dev/net/http#Client) to send the request, so we create the client as default http.Client object with defaults and call the [Do](https://pkg.go.dev/net/http#Client.Do) method with the `req` as the request parameter in order to send the request with the default client. This method returns the response object, and an error if any. We also add the `defer resp.Body.Close()` in order to avoid leaks and safely access the response body. ```go client := &http.Client{} resp, err := client.Do(req) if err != nil { log.Fatal(err) } defer resp.Body.Close() ``` At this point, we can now start consuming the response and use it for further processing as per the needs. ### Unmarshalling the Response We first read the response into a string or byte representation using the io.ReadAll method and then use the json.Unmarshal to convert the bytes into golang object/instance. ```go var updatedPost Post respBody, err := io.ReadAll(resp.Body) if err != nil { log.Fatal(err) } // convert the response json bytes to Post object in golang err = json.Unmarshal(respBody, &updatedPost) if err != nil { log.Fatal(err) } fmt.Println(updatedPost) fmt.Println(updatedPost.Title) ``` In the above example, we have read the response Body which can be accessed as the `Body` field in the [Response](https://pkg.go.dev/net/http#Response) object via the `resp` variable. So, the function will return the `respBody` as a string or an error if any. Then using this string object, we can use the json.Unmarshal function to send this string and create this `updatedPost` object of Post struct. This method will mutate this object as we have passed it by reference indicated by `&updatedPost` . So, this will do two things, one update / mutate the `updatedPost` instance from the `respBody` and give any error if any arrises during the [deserialsation](https://developer.mozilla.org/en-US/docs/Glossary/Deserialization) of the response . Now, we have the object in golang from the response bytes, we can use it as per requirements. So, that is the example in the entirety. Let’s simplify the steps which are similar to the POST/PUT method as well * Define/construct URL * Marshal the object into JSON string as the request body * Construct the request object (method, URL and the body) * Send the request with the default client * Read the response and deserialise/unmarshall * Access the object in golang ```go package main import ( "encoding/json" "fmt" "io" "log" "net/http" "strings" ) type Post struct { ID int `json:"id,omitempty"` Title string `json:"title,omitempty"` Body string `json:"body,omitempty"` UserId int `json:"userId,omitempty"` } func main() { // define URL to hit the API baseURL := "https://jsonplaceholder.typicode.com" postId := 4 postURL := fmt.Sprintf("%s/posts/%d", baseURL, postId) // define the body -> with the field to update reqBody := `{"body": "new body"}` fmt.Println("New body:", reqBody) // send a new request, with the `PATCH` method, url and the body req, err := http.NewRequest("PATCH", postURL, strings.NewReader(reqBody)) if err != nil { log.Fatal(err) } // set the header content type to json req.Header.Set("Content-Type", "application/json") client := &http.Client{} resp, err := client.Do(req) if err != nil { log.Fatal(err) } defer resp.Body.Close() fmt.Println("Response status code:", resp.StatusCode) fmt.Println("Response Status:", resp.Status) var updatedPost Post respBody, err := io.ReadAll(resp.Body) if err != nil { log.Fatal(err) } // convert the response json bytes to Post object in golang err = json.Unmarshal(respBody, &updatedPost) if err != nil { log.Fatal(err) } fmt.Println(updatedPost) fmt.Println(updatedPost.Title) } ``` ```plaintext New body: {"body": "new body"} Response status code: 200 Response Status: 200 OK {4 eum et est occaecati new body 1} title: eum et est occaecati body: new body id: 4 user id: 1 ``` As you would see, it has only updated the `body` and has not updated the other fields. If you would have sent a similar body with a `PUT` method, the results would have been different. That would have been dependent on the implementation of the API of course, but if there is only a few fields in the request body for a PUT method, it would have replaced the values with the empry values which are not present in the request body. That is the difference between a `PUT` and a `PATCH` method, the `PATCH` method, ideally should only update the fields of the entity which are mentioned in the request body, whereas the `PUT` method has to update the entire resource whether the fields are provided or not. Again, the implementation of these API on the server plays a vital role in how the behaviour defers and the method in itself would perform. This is also called as `JSON Merge Patch` ## JSON Merge PATCH The above API is implementing a [Merge PATCH](https://datatracker.ietf.org/doc/html/rfc7386) which is to say, merge the changes in the actual entity. Let’s say there is a Blog post Entity on a Server, you have a post that you are writing as an author. The post has a id of `4` let’s say and you are constantly changing the body of the post. So, you don’t want to send the `title` or `author_id` or any field that is not changing from the client again and again while saving, so the `MERGE PATCH` endpoint will be helpful in that case, where the client only sends the required fields to be updated. In this example, the user would only send the `body` of the post to the API every time it makes changes or saves the draft. In some cases, it might also want to change the title, so it will include the title, but not all the fields. The API will know it is a `PATCH` request and the content type is `json` so it will only change or update the fields that are provided in the request body to the actual entity in the database or wherever it is stored on the server. So, that is what is the JSON Merge PATCH or Merge PATCH in general. JSON Merge PATCH is specific to the JSON based document APIs. Below is the example, the same steps but a different endpoint. A user API that I have specifically created for demonstrating the difference in a PUT vs Merge PATCH vs JSON PATCH requests. ```go package main import ( "encoding/json" "fmt" "io" "net/http" "strings" ) type User struct { ID int `json:"id,omitempty"` Name string `json:"name,omitempty"` Email string `json:"email,omitempty"` Roles string `json:"roles,omitempty"` } func main() { baseURL := "https://dummy-json-patch.netlify.app/.netlify/functions" userID := 2 apiURL := fmt.Sprintf("%s/users/?id=%d", baseURL, userID) userObj := User{ Name: "dummy name", Roles: "dummy role", } var jsonPatchBody []byte jsonPatchBody, err := json.Marshal(userObj) if err != nil { panic(err) } // OR directly define the json as string //jsonMergePatchBody := `{ // "name": "new dummy name", // "roles": "new dummy role" //}` req, err := http.NewRequest("PATCH", apiURL, strings.NewReader(jsonMergePatchBody)) req.Header.Set("Content-Type", "application/json") client := &http.Client{} resp, err := client.Do(req) if err != nil { panic(err) } defer resp.Body.Close() var user User resBody, err := io.ReadAll(resp.Body) fmt.Println(string(resBody)) if err != nil { panic(err) } err = json.Unmarshal(resBody, &user) if err != nil { panic(err) } fmt.Println("Updated/Patched User", user) fmt.Println("Id:", user.ID) fmt.Println("Name:", user.Name) fmt.Println("Email:", user.Email) fmt.Println("Bio:", user.Bio) fmt.Println("Roles:", user.Roles) } ``` Original User with id 2 ```plaintext {"id":2,"name":"dummy","email":"dummyyummy@user.com","bio":"empty bio","roles":"read"} Id: 2 Name: dummy Email: dummyyummy@user.com Bio: empty bio Roles: read ``` Output of the program ```plaintext Request Body: {"name":"dummy name","roles":"dummy role"} {"id":2,"name":"dummy name","email":"dummyyummy@user.com","bio":"empty bio","roles":"dummy role"} Updated/Patched User {2 dummy name dummyyummy@user.com empty bio dummy role} Id: 2 Name: dummy name Email: dummyyummy@user.com Bio: empty bio Roles: dummy role ``` In the above example, the only fields that will be updated are `name` and `roles` , since the API is implemented to only update the fields provided in the json merge patch document (request body). As you can see that, only the `name` and `roles` are changed. The name was `dummy` that changed to `dummy name` and role changed from `read` to `dummy role` . Now, let’s see the same request but with PUT method on it. Before we hit this API however, let’s note what the user with id 2 is ```json { "id": 2, "name": "dummy name", "email": "dummyyummy@user.com", "bio": "empty bio", "roles": "dummy role" } ``` This is the result of our recent patch request. Now, we will send a PUT request to the same user with a different body. ```go package main import ( "encoding/json" "fmt" "io" "net/http" "strings" ) type User struct { ID int `json:"id,omitempty"` Name string `json:"name,omitempty"` Email string `json:"email,omitempty"` Bio string `json:"bio,omitempty"` Roles string `json:"roles,omitempt"` } func main() { baseURL := "https://dummy-json-patch.netlify.app/.netlify/functions" userID := 2 apiURL := fmt.Sprintf("%s/users/?id=%d", baseURL, userID) userObj := User{ Name: "not a dummy name", Roles: "not a dummy role", } var jsonPatchBody []byte jsonPatchBody, err := json.Marshal(userObj) if err != nil { panic(err) } fmt.Println("Request Body:", string(jsonPatchBody)) //jsonPatchBody := `{ // "name": "dummy", // "roles": "new dummy role" //}` req, err := http.NewRequest("PUT", apiURL, strings.NewReader(string(jsonPatchBody))) req.Header.Set("Content-Type", "application/merge-patch+json") client := &http.Client{} resp, err := client.Do(req) if err != nil { panic(err) } defer resp.Body.Close() var user User resBody, err := io.ReadAll(resp.Body) fmt.Println(string(resBody)) if err != nil { panic(err) } err = json.Unmarshal(resBody, &user) if err != nil { panic(err) } fmt.Println("Updated/Patched User", user) fmt.Println("Id:", user.ID) fmt.Println("Name:", user.Name) fmt.Println("Bio:", user.Bio) fmt.Println("Email:", user.Email) fmt.Println("Roles:", user.Roles) } ``` Output: ```plaintext Request Body: {"name":"not a dummy name","roles":"not a dummy role"} {"id":2,"name":"not a dummy name","email":"","bio":"","roles":"not a dummy role"} Updated/Patched User {2 not a dummy name not a dummy role} Id: 2 Name: not a dummy name Bio: Email: Roles: not a dummy role ``` As you can see the `name` and `roles` are updated, however the `bio` and `email` fields are empty. Since we only said to update the `name` and `roles` fields, but it was a `PUT` request, it expects all the fields, and if any of the fields are missing, it will consider them as empty and update those as well. So, the difference might be crystal clear now. When to use `PATCH` and when to avoid `PUT`. * When you have a large set of updates, preference could be given to PUT * If you have very specific fields to update and a very limited fields PATCH is recommended There is other type of PATCH specifically designed for JSON APIs, or I should say JSON Documents APIs. ## JSON PATCH The [JSON PATCH](https://datatracker.ietf.org/doc/html/rfc6902/) is a specification in which we can define what operations to perform on which fields, or path of the fields to replace, move or copy to. > A JSON Patch document is a JSON document that represents an array of objects. Each object represents a single operation to be applied to the target JSON document. As it takes this operations, it applies them sequentially and hence it won’t replace all the fields for the entity, as that is the expected behavior of the PATCH method. In other words, it would only apply changes to the fields and the related fields provided in the json patch document (request body). There are a few operations that you can perform with this json patch method, and provide the instructions accordingly for individual operations in the JSON PATCH document. Operations * add * remove * replace * move * copy * test So, for each of the operations, a high level definition can be considered as: * To add a field you can specify the operation as `add` , the path as the field to be added, and the value as the actual value to be added * To remove a field, you can specify the operation as `remove` , and the path as the field to remove * To replace a field, you can specify the operation as `replace`, the path as the field to be updated/replaced, and the value of the actual value to be added * To move a field, you can specify the operation as `move`, the **from** as the field to be updated/moved from and the path to the field the from value should be moved to. * To copy a field, you can specify the operation as `copy`, the from as the field to updated/copied from and the path to the field to which the value should be copied to. * The test operation is a bit different as it is used for comparison of a `path` value to the actual value specified in the object. It might return true or false, but not actually return it might be used as a checkpoint for continuing with the operation in the document. In this example, we are creating a similar patch request, but using this json patch document kind of structure. ### Construct the json-patch document ### Send the request ```go package main import ( "encoding/json" "fmt" "io" "net/http" "strings" ) type User struct { ID int `json:"id"` Name string `json:"name"` Email string `json:"email"` Bio string `json:"bio"` Roles string `json:"roles"` } func main() { baseURL := "https://dummy-json-patch.netlify.app/.netlify/functions" userID := 2 apiURL := fmt.Sprintf("%s/users/?id=%d", baseURL, userID) jsonPatchBody := `[ { "op": "replace", "path": "/name", "value": "new dummy name" }, { "op": "replace", "path": "/roles", "value": "new dummy role" }, ]` req, err := http.NewRequest("PATCH", apiURL, strings.NewReader(jsonPatchBody)) req.Header.Set("Content-Type", "application/json-patch+json") client := &http.Client{} resp, err := client.Do(req) if err != nil { panic(err) } defer resp.Body.Close() var user User resBody, err := io.ReadAll(resp.Body) fmt.Println(string(resBody)) if err != nil { panic(err) } err = json.Unmarshal(resBody, &user) if err != nil { panic(err) } fmt.Println("Updated/Patched User", user) fmt.Println("Id:", user.ID) fmt.Println("Name:", user.Name) fmt.Println("Bio:", user.Bio) fmt.Println("Email:", user.Email) fmt.Println("Roles:", user.Roles) } ``` Original User (id=2) before the request ```json { "id": 2, "name": "dummy name", "email": "dummyyummy@user.com", "bio": "empty bio", "roles": "dummy role" } ``` Output of the Program (JSON PATCH Request) ```plaintext {"id":2,"name":"new dummy name","email":"dummyyummy@user.com","bio": "empty bio", "roles":"new dummy role"} Updated/Patched User {2 new dummy name dummyyummy@user.com empty bio new dummy role} Id: 2 Name: new dummy name Email: dummyyummy@user.com Bio: empty bio Roles: read ``` ### References: * [Wikipedia: HTTP Request Methods](https://en.wikipedia.org/wiki/HTTP#Request\_methods) * [Ruby on Rails: Patch is the new primary HTTP method for updates](https://rubyonrails.org/2012/2/26/edge-rails-patch-is-the-new-primary-http-method-for-updates) * [RFC 5789](https://datatracker.ietf.org/doc/html/rfc5789) https://www.meetgor.com/posts/understanding-go-s-weak-package Go's weak package provides weak references for garbage collection. Thu, 02 Jan 2025 00:00:00 UTC Go's weak package provides weak references for garbage collection. https://www.meetgor.com/posts/golang-weak-package Introduction What is weak package about? What are weak pointers Difference of weak and strong references Context of Garbage collector and unique pointers Mechan Thu, 02 Jan 2025 00:00:00 UTC ## Introduction ## What is weak package about? ## What are weak pointers ## Difference of weak and strong references ## Context of Garbage collector and unique pointers ## Mechanism of weak pointers ## Usage of weak pointers ## Examples ## Quirks and Scenarios ## Conclusion https://www.meetgor.com/posts/2024-review Introduction It is the end of the year, and I write this yearly as I have been writing it since . I get a lot of insights on the things after writing them. Let� Tue, 31 Dec 2024 00:00:00 UTC ## Introduction It is the end of the year, and I write this yearly as I have been writing it since [2022](https://www.meetgor.com/tag/developer-life/). I get a lot of insights on the things after writing them. Let’s dive in for my journey from intern to full-time backend developer. > “2024, You had me in the first half” No jokes here, I was scared as shit till May 31st, something special happened on May 31st, 2024 in my life! Well, I got a full-time offer letter and the results of my bachelor’s degree which I passed with an 8.9 CGPA (not good but not bad either). If someone had to tell me an hour could change your life then this would be the day, exactly between 5 and 6 in the evening, I went from dropped shoulders to the happiest person in the world. I can’t describe that feeling; it has a special place in my heart. From June 2023 to May 2024, I faced rejections left and right, but there was hope, a flicker of fire left that carried me here. I went from having nothing to being overwhelmed by so many great things that happened to me. I am grateful to my parents, family, manager, friends, and Mother Nature for blessing me with whatever I receive in 2024 🥹 ### Gist It was a good year, I learned a lot. I can say confidently I am more mature than I was a year ago. The year 2023 was a bit harsh for me, especially in the end, but that continued the effects for some time in 2024 as well. However, as I said, that single hour changed my life in a good way. I became more of a problem thinker than a software developer, I was solving real-life problems at work. I faced challenges and failed at times, but I reflected, learned from them, and moved ahead. This is the aim of this post as well. I want to learn from this year. It feels overwhelming at first, but if I put it word by word, sentence by sentence, then it becomes crystal clear where I shined and where I fumbled, and the holes in the wall. I feel like the first half of 2024 was the test of my patience and consistency. I was interning at Docsumo, I was assigned the task of one specific feature, then I got a few more tasks simultaneously and kept on chipping in bug fixes and understanding of the codebase, which by the end of one quarter I was given the sole responsibility of maintenance. That was not as boring as you might expect, it was challenging, it was about thinking through edge cases and solving the issues that the users were facing. This helped me develop my problem-solving skills and improve my communication skills. ## Lessons learned - Hard work pays off, it might take time, but it does, patience is the key - Things can change in an instance, don't wait for the right/perfect time, make every second count - People are going to judge anyway, so don't hold back on the things that you wish to do, and don't let that fear hold you - Writing is one of the things that can make your thoughts and impact immortal (to some extent) I won't leave this article without a quote or two, I really think quotes can change your perspective of life (not all obviously, but some) > "Do one thing every day that scares you" > -- Eleanor Roosevelt I started recording videos, recording the voice-over (yikes, I hated my voice for the initial few videos, I still hate it, but have gotten used to it). I don't just say, I do! Or I do and then say like this article 😎 One more, please? > "It's not the years in your life that count. It's the life in your years." > -- Abraham Lincoln This is how I feel right now, I overcame one obstacle in my life, and the other is right here waiting for me to shoot down. You can't wait for life to ease out for you, you will have to find happiness in the hardships and highs and lows in your life. ## Achievements * Land a full-time offer as a junior backend developer at [Docsumo](https://www.docsumo.com/) * Started a YouTube channel, despite fear and doubts * Started a [Tech Newsletter](https://techstructively.substack.com/) (summary of my learnings) with 23 editions written in 23 weeks * Created a writing routine for 50 days (wrote some literature/philosophical-related articles) I feel good, but there is a mountain to climb. It is a marathon, not a sprint. I am excited about the year ahead, all the things I will learn, what responsibilities will be given, and will I make new connections in tech. ## Technical Retrospective I would like to view the year into the things I created namely the blogs and the video that I made. ### Articles Written This could have been better but still managed to write around 13 articles (excluding this), about Cody setup in Neovim, Nginx basics, Golang HTTP POST and PUT methods, Advent of Code approach and solutions in Golang. * [Configuring Sourcegraph’s Cody in Neovim](https://www.meetgor.com/neovim-sourcegraph-cody/) * [NGINX Survival Guide](https://www.meetgor.com/series/nginx-survival-guide) * [100 Days of Golang](https://www.meetgor.com/series/100-days-of-golang): Edition 34th and 35th Article about [POST](https://www.meetgor.com/golang-web-post-method/) and [PUT](https://www.meetgor.com/golang-web-put-method/) method * Using LibSQL Database in Golang: [Connection](https://www.meetgor.com/turso-libsql-db-golang/) and using [Embedded Replicas](https://www.meetgor.com/turso-libsql-embedded-replicas-golang/) * [Safely using Maps in Golang: The difference between declaration and initialization](https://www.meetgor.com/golang-safely-using-maps/) * [Advent of Code, 2024 in Golang](https://www.meetgor.com/series/advent-of-code-2024): Approach and Solution Explanation (till day 5, might write a few more, have some rough drafts to be completed) ### Youtube Channel On September 9th I published my first YouTube video, I always wanted to create some tutorial about the things I know or want to learn. On that day, I kept my doubts and fears aside and hit the record button. It went bad, but I published the video. It took me roughly 5 hours to record the voice-over for a 6-minute video, I was in shambles after that. I then decided to double down on creating YouTube videos, to remove my fear of communicating, speaking, and expressing ideas and emotions through YouTube. And after 4 months, I have published around 14 videos. Not only that, I started to do live streams on weekends as well. I did around 16 live streams on Twitch and YouTube, I feel a lot better and a lot more confident. And yes the time required for me to create a video, record voice-over, and do the editing is significantly improving with each video, the last video I posted of 50 minutes, I recorded within an hour, one shot, and the next hour, the video was published. If I am the only person to benefit from my videos, then so be it. Here is the summary of the videos I made from September to December: * Using Turso LibSQL Database in Golang \[[remote-connection](https://youtu.be/vBrvX0X0phw?si=eEDV_qa9uXqI_vyK)\] \[[embedded replicas](https://youtu.be/BitxB40rdVw?si=E3KK1hoGf1hY0klq)\] \[[libsql local server](https://youtu.be/rIXePHhdUSI?si=IYk9A4zpBvr9pILf)\] * Using [Goose](https://youtu.be/9kMeb7d2id8?si=FHyFKcXme8WyFij_) and [SQLC](https://youtu.be/mv1vCqUy_5o?si=dG25tC1U_zsoAhPd) with LibSQL in a Golang Application * [Deploying Netlify cloud functions in Golang](https://youtu.be/BY2Z2Em7OdA?si=u2zjCWoOOI4wRivC) * What’s new in Golang 1.24 \[[SplitSeq](https://youtu.be/hee9KUhvQsY?si=FqhGRXbhKOgPrVLj)\] \[[Omit Zero JSON Tag](https://youtu.be/RFUz4_axOZg?si=88iUnjGBKCx_fFiv)\] * Advent of Code \[[day-1](https://youtu.be/4U97gLyz0Ss?si=vxfmbQxaaUf1GRmk)\] \[[day-2](https://youtu.be/jGrPkldp4rY?si=Brdb1ulBYwUfXr7F)\] \[[day-3](https://youtu.be/nhueQbOKdTs?si=MjHPXbKoCml5x-Ut)\] \[[day-4](https://youtu.be/029R_DKLqko?si=Iv5rVD_meKlXbMJs)\] \[[day-5](https://youtu.be/7pl7WRmfepQ?si=QdGigfhn04rSUEp4)\] … ran out of time I just have only 50 odd subscribers and a couple of thousand views, it is just the start, and I am not looking into the metrics right now, surely I am trying to improve and build a community, but numbers don't mean anything to me right now at least. I am creating videos that I would like to watch myself if I hadn't known about the concepts that I once didn't knew. ### Open Source Contributions I wanted to do many more open-source contributions, but since I worked on creating YouTube videos, I didn’t get much headspace to work on them. Next year I want to create some form a habit to read some code, that way I know I will bang my head to fix the issues or discover something new. Creating a chain of exploring, learning, and contributing. * Minor additions on [steampipe](https://github.com/turbot/steampipe/pull/4315) and [turso-cli](https://github.com/tursodatabase/turso-cli/pull/815) repositories - Added CSV flag for separator in db create command - Fix the bug for only showing timing when verbose flag is set ### Projects Created I created a few projects, some of them just to explore the technologies and little MVPs, one project that I can be open to showing is the [Outsmarty](https://github.com/Mr-Destructive/outsmarty) project. This project is like a Psyche Clone in Golang, it’s not a mobile application, it is a web app. I have not completed it as usual but it is a good concept that I created till most of the heavy lifting, I usually quit at the frontend interfaces. ## Non-Technical Stuff I was really into writing and reading in the second half of the year. I decided to create a morning routine for a couple of weeks, but Indian parents, had to put that routine into trash can, but it was a good experience while it lasted. * Read 6 books \[[GoodReads profile](https://www.goodreads.com/user/show/82596574-meet)\] * Wrote [50 Posts](https://mythstomeaning.substack.com/) about mythology and philosophy Hopefully, as I carry forward this momentum and energy, I will likely have one novella completed this year. I not just hoping to do that I am planning it now. ## Goals for 2025 Moving into 2025, I want to write more code and of course articles and videos. Let’s get more specific of what I am thinking of: * Write blogs for FreeCodeCamp (at least 3) * Learn Kubernetes in detail * Write 25 articles on 100-days-of-Golang * Learn Rust or Zig * Get 2K subscribers on YouTube (100 videos) ## Conclusion It was a year to remember for sure. I said this last year as well but in a bad way. If there was a year to forget, it would have been late 2023 and the first half of 2024, but the second half of 2024 was a blessing for me. I overcame my fear of speaking to some extent, at least starting to address the issues. If I were to sum up the year, I am definitely happier than I was last year, and a lot more excited than I was last year. Money can't buy happiness, if you don't find joy within, nothing can be happiness. I think curiosity, consistency, and grit are pillars of my life and I stick to it this year and want to continue to do so. I will leave you with one more quote, > "Life is not about waiting for the storm to pass, it's about learning to dance in the rain." > -- Vivian Greene I hope your year was good too, if not may 2025 be the year of your dreams, you absolutely got this. See you on the other side 👋 Happy New Year 🎆 And as always, Happy Coding :) https://www.meetgor.com/posts/aoc-2024-day-5 Introduction It is day 5 of the advent of code, and today we have an interesting problem of ordering pages. Let’s dive into the problem and how I approached i Wed, 11 Dec 2024 00:00:00 UTC ## Introduction It is day 5 of the advent of code, and today we have an interesting problem of ordering pages. Let’s dive into the problem and how I approached it. It was a pretty simple problem if thought it peacefully, otherwise, it would get into a map, list, and indices mess. ## Input In the input for day 5, we have two sections, The first defines the rules for ordering the pages, specifically which page should come before which and the second contains the actual order of pages. ```plaintext 47|53 97|13 97|61 97|47 75|29 61|13 75|53 29|13 97|29 53|29 61|53 97|53 61|29 47|13 75|47 97|75 47|61 75|61 47|29 75|13 53|13 75,47,61,53,29 97,61,53,29,13 75,29,13 75,97,47,61,53 61,13,29 97,13,75,29,47 ``` So, the first section has the rules mapped out, the other has the ordering of pages, and each line is a query or a list of pages as our actual data to process. We need to use it in the processing of parts 1 and 2. ### Reading Sections So, we need to parse these sections and read them in a data structure that could be easier to access. One way to do that would be * A list with two sections * The first section will be a list * The list will be a list of integers to hold the two integers i.e. for rules * The second section will be a list * The list will be a list of integers to hold the page list So, the data structure would look like a list of list of list of integers. ```go func ReadFileSections(path string) [][][]int { fileBytes := ReadFileBytes(path) lines := []string{} separator := []byte(" ") for _, line := range bytes.Split(fileBytes, separator) { if string(line) != "" { lines = append(lines, string(line)) } } sections := [][][]int{} for i, section := range lines { nums := [][]int{} lineStrs := strings.Split(section, " ") separator := "," if i == 0 { separator = "|" } for _, lineStr := range lineStrs { if lineStr == "" { continue } numL := []int{} for _, numStr := range strings.Split(lineStr, separator) { num, _ := strconv.Atoi(numStr) numL = append(numL, num) } nums = append(nums, numL) } sections = append(sections, nums) } return sections } ``` The above function called `ReadFileSections` takes in a path to the input file and returns a slice/array) of the list of list of integers as discussed. We first read the file and split the bytes into two newline characters that will be the separator for the sections, we will store the lines as a list of strings, the first will contain the rule lines and the second will contain the page list lines. Then we iterate over the section and split the individual lines for sections separately with the respective separator i.e. `|` for the first section and (whitespace) for the second section. We are parsing each line to get a list of integers and append them to the respective sections. So, we now have data that we can use to construct the rules and pages to help process the problem. ### Constructing Rules Now, we need to process the rules list for convenient access, we need to get the page number that should appear after a given page, so we will use a map of integer with a list of integers, where the key will be the first number and the one of the value will be the second number ( the number that should appear after in order of the pages). ```go func ConstructRules(rulesList [][]int) map[int][]int { rules := make(map[int][]int) for _, rule := range rulesList { rules[rule[0]] = append(rules[rule[0]], rule[1]) } return rules } ``` We simply iterate over the list of integers and map the first element as the key and the value as the second element in the list, so to visualize: ```go FROM [][]int [ [47,53] [97,13] [97,61] ] TO map[int][]int { 47: [53] 97: [13,61] } ``` So, now have the rules as a map of integers with integers. ### Constructing indices Now, to make the first and second parts easier, we need to make a map for each number in the rules section with the indices that appear in the pages list. So, we will iterate over the rules, which is a map of integers and integers, we will create a map of integers that will help us create a unique list of integers from the rules. Now, once we have the list of integers from the rules, we will iterate over all the numbers and on each page line, check on which index it appears, for creating a list of integers(indices). So, we iterate over all the numbers in the line of pages, if we find that number in the list of pages, we append the index, however, if we don’t we append -1, so for each line we need to have an index appended for that number like so: ```plaintext # 75 75,47,61,53,29 -> 0 97,61,53,29,13 -> -1 75,29,13 -> 0 75,97,47,61,53 -> 0 61,13,29 -> -1 97,13,75,29,47 -> 2 75[0,-1,0,0,-1,2] # map[int][]int # 75 -> int # [0,-1,0,0,-1,2] -> []int ``` So, in the above example, we have taken 75 for reference, we will get the index for each list of page numbers, and we get the list of indexes where 75 appears. Now, this can be done with the following function: ```go func GetPageIndices(rules map[int][]int, pages [][]int) map[int][]int { nums := make(map[int]bool) for num, list := range rules { nums[num] = true for _, elem := range list { if !nums[elem] { nums[elem] = true } } } numIndices := make(map[int][]int) for num, _ := range nums { for _, numLine := range pages { index := -1 for i, n := range numLine { if n == num { index = i } } numIndices[num] = append(numIndices[num], index) } } return numIndices } ``` So, we now have the index mapped at each page numbers list from the rules. ## Part 1 Now, for part one we need to iterate over each page update (line), then we need to check if the page numbers are following the rules, each number should follow the rules. This means that if a number is after a certain number but the rule says it should be before, then it has violated the page numbering rule in that update, so we cannot consider it as the correct ordered page, we need to add the middle page number of each update which is correctly ordered as the answer for the part one. To do that, we iterate over each page update, then we need to iterate over each of the numbers in that page update, we obtain all the rules associated with that number (let’s call it the current number) since we have a map of integers with a list of integers. Now, we have to check if the number that we currently are in is before the numbers in its rules. So, we check with the index of the current number using the number indices that we created which is a map of the number with a list of integers as indices. So, we obtain the list of indices of the map with the current number as the key for the map and the index in the list as the number of line/page updates we are currently in. Then once we have got the index for the current number, we obtain the same for the second number which is all the numbers in its rule, and if that number in its rule is present in that page line/update i.e. it is not -1 and if that is the case, we get the index of it similarly and check if it appears after the current number following the rule, And so if any number violates the rule, we need to mark the page update as not in correct order. As we see that the index rule for that page update is violated, we mark the order as false. If we see the ordered flag is still true, we update the score with the middle element of that page update. ```go func GetOrderedPages(rules, numIndices map[int][]int, pages [][]int) int { score := 0 for index, pageLine := range pages { ordered := true for _, num1 := range pageLine { rule := rules[num1] index1 := numIndices[num1][index] for _, num2 := range rule { index2 := numIndices[num2][index] if index1 == -1 || index2 == -1 { continue } if index1 > index2 { ordered = false } } } if ordered { score += pageLine[int(len(pageLine)/2)] } } return score } ``` So, to reiterate, we create a function called `GetOrderedPage` with rule and number indices as a map of integers with a list of integers and the pages which is a list of integers as the page update. We return the score as the output of this function. We iterate through each of the page updates, then through each page number in the update, we check for the rule of that number and if the index of that is lower than the current number we mark it as not ordered, and hence at the end of each page update we update the score with the middle element of the page update, if the order is correct. So, that will be part one summed up, we just have to get the score of the correct ordered page updates. ## Part 2 In part 2 however, we need to check if the page update is in order, if it is not then we need to make it in order. We do a similar thing for part 2, we need to iterate over each of the page updates, and for each number in that page update, we need to check if the rule is violated or not, if we encounter any case where the rule is violated for any number, we mark the flag ordered as false, this we will use to correct the order of the page updates. After updating the pages in that page line/update, we need to add the score with middle element of the corrected order of page update. ```go func GetCorrectOrderedPages(rules, numIndices map[int][]int, pages [][]int) int { score := 0 for index, pageLine := range pages { ordered := true for _, num1 := range pageLine { rule := rules[num1] index1 := numIndices[num1][index] for _, num2 := range rule { index2 := numIndices[num2][index] if index1 == -1 || index2 == -1 { continue } if index1 > index2 { ordered = false } } } if !ordered { newLine := CorrectPageOrder(pageLine, rules) score += newLine[len(newLine)/2] } } return score } ``` We need to implement the CorrectPageOrder function that takes in the page line or page update and the rules, we need to create a new page update, that will populate the page that follows all the rules. So, we first keep track of the initialized elements index and update the index if we need to move the element before it. So, we iterate over all the numbers in the page update and set the index before any number in the rule, if we encounter any such number in the rule map, we need to update the index with the index of that number. And once we have got the index where we want to swap the element to, we create a slice before that index and append that number into it, and append everything after that index. ```go func CorrectPageOrder(line []int, rules map[int][]int) []int { newLine := []int{} for _, num := range line { index := make(map[int]int) for i, n := range newLine { index[n] = i } newInsertIndex := len(newLine) for _, rule := range rules[num] { if idx, ok := index[rule]; ok { if newInsertIndex > idx { newInsertIndex = idx } } } afterNum := slices.Clone(newLine[newInsertIndex:]) newLine = append(newLine[:newInsertIndex], num) newLine = append(newLine, afterNum...) } return newLine } ``` So, this function will find the index of a number to place it at the most extreme left (beginning of the list) so that we are not violating any rules for that number, then we create a slice to append that number before that index and append everything after that index. That’s it from part two, we have updated the page order if there were any discrepancies in the page order. ## Conclusion So, that is it from day 5 of Advent of Code in Golang, let me know if you have any suggestions, and how you approached it. any better solutions? Happy Coding :) https://www.meetgor.com/posts/aoc-2024-day-4 Introduction Moving on to day 4, we have a grid problem in front of us, we are given some numbers in the form of a grid, i.e. some rows and columns with some up Mon, 09 Dec 2024 00:00:00 UTC ## Introduction Moving on to day 4, we have a grid problem in front of us, we are given some numbers in the form of a grid, i.e. some rows and columns with some upper case letters. What we need to do is to find is the word `XMAS` in any direction (up, left, down, right, diagonals), and in the second part we need to find the word `MAS` forming an X. So, let’s see how we can approach this and solve it in golang. You can check out my solutions [here on GitHub](https://github.com/Mr-Destructive/advent_of_code/blob/main/2024/src/day04/main.go)[.](https://github.com/Mr-Destructive/advent_of_code/blob/main/2024/src/day04/main.go) ### [Constructin](https://github.com/Mr-Destructive/advent_of_code/blob/main/2024/src/day02/main.go)g the grid The most fundamental part of the problem lies in actually converting the text into a grid or a matrix form. We can split the lines, into individual lines and append each character as an element in a list, and that way we can have a list of list of strings which is a matrix or grid-like (2-dimensional) structure. So, below is the input for the puzzle. ```plaintext MMMSXXMASM MSAMXMSMSA AMXSXMAAMM MSAMASMSMX XMASAMXAMM XXAMMXXAMA SMSMSASXSS SAXAMASAAA MAMMMXMMMM MXMXAXMASX ``` We need to convert it into something like this ```plaintext [ [M M M S X X M A S M] [M S A M X M S M S A] [A M X S X M A A M M] [M S A M A S M S M X] [X M A S A M X A M M] [X X A M M X X A M A] [S M S M S A S X S S] [S A X A M A S A A A] [M A M M M X M M M M] [M X M X A X M A S X] ] ``` So, this is a list of strings, we can say in golang it is a `[][]string` . We can do that by creating a function like this: ```go func ConstructGrid(lines []string) [][]string { grid := [][]string{} for _, line := range lines { row := []string{} for _, char := range strings.Split(line, "") { row = append(row, char) } grid = append(grid, row) } return grid } ``` The above function takes in a list of strings and returns a list of list of strings that are individual letters in the grid. We can read the file bytes and split the bytes on newline characters and then this will be used as the input for this function. So, once the input is parsed into a grid, we can start thinking about the actual logic of finding the word `XMAS` in it. ## Part 1 So, in the first part, we need to find the word `XMAS` in the matrix which could be appearing: * forwards (as `XMAS`) * backward (as `SAMX`) * upwards ```plaintext S A M X ``` * downwards ```plaintext X M A S ``` * Diagonal upwards (right or up left) ```plaintext S A M X OR S A M X ``` * Diagonals downwards (right or left) ```plaintext X M A S OR X M A S ``` So, there are 8 directions where `XMAS` could appear in the grid, there could n number of these `XMAS` . We need to find the count of these in the grid.  To approach this, we can either find the first character in the word `XMAS` and then search in all directions one by one and check if we find `M` and if we have found `M` in any of the direction, we keep moving ahead in that direction and check if there is `A` and `S` in that direction. The approach looks like this: * Initialize the counter to 0 * Iterate over each line * Iterate over each character in the line * Check if the character is equal to `X` * If the character is `X`→ * Iterate over all the directions (up, down, right, left, up-left, up-right, down-left, down-right) * For that direction if we find the character to be `M` * Keep moving ahead in the same direction to find `A` and `S` similarly, if we found all the characters `XMAS` then, increment the counter * Else choose another direction in the loop This looks complex and large but is simple, focus one thing at a time and you can solve this easily. So, for the implementation of this, we need to define a few things first: ```go var directions [][]int = [][]int{ []int{0, -1}, // up []int{0, 1}, // down []int{1, 0}, // right []int{-1, 0}, // left []int{1, -1}, // up right []int{-1, -1}, // up left []int{1, 1}, // down right []int{-1, 1}, // down left } var wordList []string = []string{"X", "M", "A", "S"} ``` So, we have defined the list of integers in the directions which are the x and y coordinates that we need to add or subtract to get to the desired location. It is basically like a unit vector, it has a distance of 1 and a direction indicated by `+` or `-` indicating to move the left or right for x coordinates and up and down for y c-ordinates. So, let me explain that more clearly, let’s say I am at `(1,2)` in the grid which is of 4x4 dimension. ```plaintext A B C D E F G H I J K L M N O P ``` So, at 2,1 we have `G` , so we check some directions for this up → `0,-1` → 2+0, 1-1 → 2,0, we have moved to `C` right → `1,0` → 2+1, 1+0 → 3,1 , we have moved to `H` down, left → `-1,1` → 2-1, 1+1 → 1, 2, we have moved to `J` So, you get the idea, that we are moving in some directions using these coordinates. We can use these to get the next direction jump we want to make to find if that element has the next character in the word that we are searching. We will write a function that does this first and abstract the function that checks if we have found the word in the grid. ```go func TraverseGrid(grid [][]string) int { score := 0 for x, row := range grid { for y, char := range row { if char == wordList[0] { for _, direction := range directions { if FindXMAS(x, y, 1, direction, grid) { score += 1 } } } } } return score } ``` The above function takes in a grid and returns an integer which will be the score i.e. the count of words `XMAS` found in the grid/matrix. First, we need to iterate through each of the rows in the grid, for each row, we iterate over the character, so we will have x and y coordinates as the index of the grid. We need to then check if the current character is `X` or `wordList[0]` , if that is the case, we iterate over all the directions and check if we can find `XMAS` i.e. `MAS` in that direction, if so we increment the counter. What is the `FindXMAS` function, let’s abstract that away, and pass in the `x`, `y`, which are the coordinates of the current word, `1` which will be the word position of the `XMAS` and in this case, we already have found `X` we need to find `MAS` in that direction. We pass the grid and the direction, so this function will return true or false if that direction has `MAS` in it. So to iterate: * We iterate over the grid and get `row` and `x` as the list of strings and index of the current row. * For each row i.e. list of strings, we iterate over the list of strings to get `char` and `y` as the character (string) and the index of that character in the list of the string. * If we find the current character to be equal to `X` which is the 0th index of the `wordList` then * We iterate over all the directions and call the function `FindXMAS` to check if the remaining word `MAS` in that direction * If we find all the words, we increment the counter. * So, we return the counter as we count the number of words `XMAS` in the grid/matrix. Now, we can implement the `FindXMAS` function, that takes in a `x`, `y` coordinates, the `wordPosition`, the direction and the grid, and return if the word is found. * First, we take the current x coordinates and add the direction’s x component (0th index or first element) * add current y coordinates to the direction’s y component (1st index or second element) * if the word position i.e.. the word index or the word itself in the current function is equal to the wordList, it means that it has found the required word completely * We need to check by adding the direction to the x and y coordinates, we are not overshooting the width and height of the grid, so if we do, we return a false * The final if is for checking if the current character is equal to the word that we are looking for, it could be `M`, `A` , or `S` . If so, we return the recursively call the `FindXMAS` function by passing the updated x and y coordinates and the next word in the wordList, we keep the direction the same and pass the entire grid. ```go func FindXMAS(x, y, wordPosition int, direction []int, grid [][]string) bool { xNext := x + direction[0] yNext := y + direction[1] if wordPosition > len(wordList)-1 { return true } if xNext < 0 || xNext >= len(grid) || yNext < 0 || yNext >= len(grid[x]) { return false } if grid[xNext][yNext] == wordList[wordPosition] { return FindXMAS(xNext, yNext, wordPosition+1, direction, grid) } return false } ``` So, we have implemented the `FindXMAS` function, this will just return if we have found the `MAS` word by going in a particular direction by updating the coordinates and checking if the word at that position in the gird is the next word in `MAS` list. So, this is what the entire first part looks like: ```go func main() { lines := ReadFileLines("sample.txt") grid := ConstructGrid(lines) score := TraverseGrid(grid) fmt.Println(score) } ``` We take in the lines as a list of strings and pass that to `ConstructGrid` and get the grid, finally, we call `TraverseGrid` , by passing the grid and getting the score as the count of the words `XMAS` in the grid. That’s it from the part 1. ## Part 2 For part two, we need to find `MAS` in the cross shape, like below: ```plaintext M.S .A. M.S ``` So, to solve this, we can do a similar approach but much simpler, we just need to find `A` as there will always be the word `MAS` in the center, so we just check if we have `A` and the top-left, top-right, or bottom-right, bottom-left has `M` or `S` . We get the coordinates of the top-right, top-left positions, down-right, and down-left positions by adding and subtracting 1 from it. We make a basic check if we are not overshooting the boundary of the grid. If we overshoot the boundaries, we won’t find the `MAS` But if we are within the grid, we now get the character at those 4 positions, we check if the top-left and bottom-right have `M` and `S` or `S` or `M`, similarly for top-right and bottom-left has `M` and `S` or `S` or `M` respectively. This is the diagonal search for `M` and `S` above and below the character `A`. So, if we have both the diagonal matched we return true. ```go func FindMAS(x, y int, grid [][]string, wordList []string) bool { xL, yT := x-1, y+1 // Top-left neighbor xR, yD := x+1, y-1 // Bottom-right neighbor // Check if the indices are within bounds if xL < 0 || xR >= len(grid) || yT < 0 || yD < 0 || yT >= len(grid[xL]) || yD >= len(grid[xR]) { return false } topLeft := grid[xL][yT] bottomRight := grid[xR][yD] topRight := grid[xR][yT] bottomLeft := grid[xL][yD] word1, word3 := wordList[1], wordList[3] isDiagonalMatch := (topLeft == word1 && bottomRight == word3) || (topLeft == word3 && bottomRight == word1) isAntiDiagonalMatch := (topRight == word1 && bottomLeft == word3) || (topRight == word3 && bottomLeft == word1) return isDiagonalMatch && isAntiDiagonalMatch } ``` So, that is the simple implementation for finding the `MAS` the diagonal. Now, we need to change the `TraverseGrid` a bit, as we just iterate over the grid, and check if we have `A` in the character in the row, i.e. `wordList[2]`. Now, if we have `A`, we need to call the `FindMAS` function with the current coordinates and the grid, if that function returns true, we increment the counter,. ```go func TraverseGrid2(grid [][]string) int { score := 0 for x, row := range grid { for y, char := range row { if char == wordList[2] { if FindMAS(x, y, grid) { score += 1 } } } } return score } ``` So, that is the final implementation of part 2, we get the count of `MAS` in the cross direction. You can check out my solutions [here on GitHub](https://github.com/Mr-Destructive/advent_of_code/blob/main/2024/src/day04/main.go)[.](https://github.com/Mr-Destructive/advent_of_code/blob/main/2024/src/day04/main.go) ## Conclusion So, that is it from day 4 of Advent of Code in Golang, let me know if you have any suggestions, and how you approached it. any better solutions? Happy Coding :) https://www.meetgor.com/posts/aoc-2024-day-3 Introduction Well, it is day 3 of the advent of code 2024, and I have been doing it on live streams. I am behind two days but working through them one by one. S Sat, 07 Dec 2024 00:00:00 UTC ## Introduction Well, it is day 3 of the advent of code 2024, and I have been doing it on live streams. I am behind two days but working through them one by one. So far, I have learned a lot of things in Go. Let’s dive in for the day 3. ## Part 1 Part one to any AOC problem seems straightforward, but as soon as part two is revealed, the real implementation starts to break the sweat (if you weren’t optimistic or thoughtful) For part 1 for this day, was to parse a string that contained `mul(X,Y`) an expression, where X could be any 3-digit number. So, there could be multiple such expressions within the string and the purpose was to multiply the X and Y in the individual expression and add them up.  ```plaintext xmul(2,4)&mul[3,7]!^don't()_mul(5,5)+mul(32,64](mul(11,8)undo()?mul(8,5)) ``` In this example above there are 4 such expressions, and if we add the multiplications of those, we get the answer as 161. ### Approach It looks like a Regex pattern, finding an expression-like pattern in a string. So, the approach would be to find such expressions with a regex pattern and parse the numbers to integers, and multiply them, simply. You could go ahead and write the parser for iterating over each character in the string and parsing the tokens, then evaluating the expression. That is a valid approach, but I choose to do this because I don’t know how to write a parser honestly, I want to try that solution at the end as well. But for the first part, a quick regular expression seems a good idea. ### Constructing the Regular Expression The first thing is to write the regular expression for the `mul(X,Y)` part which is the only challenging section in part one. The rest is just simple math. So, we need to find `mul`, then a `(` then any number which is 1 to 3 digits long, then `,` and again a number that is 1 to 3 digits long, and finally ends with a `)` That translates to: ```plaintext mul\((\d{1,3}),(\d{1,3})\) ``` Let’s breakdown: * `mul` for capturing the literal word `mul` * `\(` this is for the first parenthesis in the expression `mul()` , we need to escape the bracket in a regular expression if we want to match it, so we use `\` before it. * Then we have a match group `(\d{1,3})` , this will be the `X` in `mul(X,Y)`: * A match group is like a group of a match in a regex, basically, if you want to capture specific parts in the entire match, then we use `()` to group them individually, this is not necessary but helps in getting the right things without overhead * In this case, we are using a match group for capturing a number which can have 1 to 3 digits. * The other way to write this is `([0-9]{1,3})` , which also would do the same thing, (NOTE: there are some differences in `[0-9]` and `\d`, but that is very subtle and won’t affect this puzzle, if you are still curious, I prefer reading this [StackOverflow question](https://unix.stackexchange.com/questions/414226/difference-between-0-9-digit-and-d)) * We then use `,` for the separator of `X` and `Y` operands in the `mul(X,Y)` expression * We then similarly do the match for `Y` in `mul(X,Y)` with the `(\d{1,3})` match group * Finally, we end the regular expression with the `)` to end the expression ### Code it This is quite straightforward, we grab the line as a string and use [regexp.MustCompile](https://pkg.go.dev/regexp#MustCompile) function which gives us a [Regexp](https://pkg.go.dev/regexp#Regexp) object, that in turn has a few methods associated with it to find, match, replace, and other things that can be done with a regular expression on a string. Once we have the `mulRegex` , we can use the [FindAllStringSubmatch](https://pkg.go.dev/regexp#Regexp.FindAllStringSubmatch) method associated with the `Regexp` struct in the `regexp` package. The function takes in a string to perform the regex on, and the maximum number of matches to return. We want all the results, so we pass them in `-1` to get all the matches. Now, this method returns a slice of a slice of strings, each slice is a match, and within each slice, there is a slice of string, with the matched string and the subsequent match groups in the string if any. ```go func FindMulExpression(line string) [][]string { mulRegex := regexp.MustCompile(`mul\((\d{1,3}),(\d{1,3})\)`) return mulRegex.FindAllStringSubmatch(line, len(line)) } ``` So, the above function will return something like this ```go [ [mul(2,4) 2 4] [mul(5,5) 5 5] [mul(11,8) 11 8] [mul(8,5) 8 5] ] ``` This is a list of list of strings, these look like numbers but those are string types in Golang. Now we have this, we can create the actual logic part of obtaining the result, which is to parse these expressions, multiply `X` and `Y` and add the results for each of the expressions. ```go func Multiply(matches [][]string) int { score := 0 for _, match := range matches { x, err := strconv.Atoi(string(match[1])) if err != nil { panic(err) } y, err := strconv.Atoi(string(match[2])) if err != nil { panic(err) } score += x * y } return score } ``` This is pretty straightforward, we iterate over each of the matches, that is one `mul(X,Y)` expression and parse the `X` and `Y` each into integers and multiply them, the result obtained is then added to the score. We do this for each `mul(X,Y)` expression that is found in the string(line) and return the final score. ### Input Now, the example gave us a single string, but I realized there were six lines in the input (individual input), so we needed to parse each line and add up the score. Remember this as it will be critical in part 2, it took me some time and questioning my existence to realize we need to combine all the lines to get the result (not necessary in part 1 but for sure in part 2). ## Part 2 This is where things go wrong usually. At least for me, it did. I started with a very naive approach, with a forever loop and finding the index of do or don’t and stripping off those sections, and then looping until we have no do and don’ts left. That was working for the test case but failed on the actual input. The approach I finally came up and was working by tweaking the same approach slightly. ### Approach What I came up with is to find the first match location of `don’()` and `do()` strings in the entire string, we take that and remove the parts after `don’t()` or before `do()` . That way we can trim down the string to only valid/enabled `mul()`expressions.  So, the approach more clearly defined will be: * Find the location (index) of the `don’t()` and `do()` expressions * We need to keep track of whether the previous string was enabled or disabled so will keep a flag to append the enabled expressions (part of the string) to the final result * If none of them are found, return the string(line) as it is * If we found either of them then: * If we find don’t first (`don’t()` appears before `do()`) * If the enabled flag is true → append the string before the `don’t()` expression * Then toggle the enabled as false and trim off the `don’t()` part (This way we have completed checking everything before the don’t expression, so we remove that part from the line string) * If we find do first (`do()` appears before `don’t()`) * If the enabled flag is true → append the string before the `do()` expression * Then toggle the enabled as true and trim off the `do()` part (This way we have completed checking everything before the do expression, so we remove the part from the line string) * We do this until there is no line string left to be checked ### Code I used simple Strings.Index to get the first match index for the substring, In this case, I want the first matching index for `don’t()` and `do()` . Once we have the indices of both the matches, we can iterate over till we are not left with any do or don’ts in the string. If we have either do or don’t we append to the string the part before don’t if enabled or part before do if enabled and toggle on and off the enabled flag accordingly. By the end of the loop, the result string will have only the enabled parts of the line/string. ```go func StripDoDont(line string) string { result := "" enabled := true dontOffset := len("don't()") doOffset := len("do()") for len(line) > 0 { dontIndex := strings.Index(line, "don't()") doIndex := strings.Index(line, "do()") if dontIndex == -1 && doIndex == -1 { if enabled { result += line } break } if dontIndex != -1 && (doIndex == -1 || dontIndex < doIndex) { if enabled { result += line[:dontIndex] } enabled = false line = line[dontIndex+dontOffset:] } else { if enabled { result += line[:doIndex] } enabled = true line = line[doIndex+doOffset:] } } return result } ``` I take this function and pass it to the multiply function where I get the matching patterns for the `mul` expression and do the math. The [strings.Index](https://pkg.go.dev/strings#Index) method takes in a string and a substring to find within that string and returns the index of the first occurring instance of that substring. With that we can identify if the line string contains the `do()` or `don’t()` expressions, if they don’t we simply return the line and if there are instances of them, we loop and trim the string before and after the expressions depending on whether the flag is enabled or disabled. Let’s take an example and walk through the logic: ```plaintext abcxmul(1,3)don't()mul(9, 7)do()mul(1,2)don't()mul(8,7) enabled = True result = "" line = "abcxmul(1,3)don't()mul(9, 7)do()mul(1,2)don't()mul(8,7)" --- After Iteration 1: result -> abcxmul(1,3) line -> mul(9, 7)do()mul(1,2)don't()mul(8,7) enabled = False --- After Iteration 2: result -> abcxmul(1,3) line -> mul(1,2)don't()mul(8,7) enabled = True --- After Iteration 3: result -> abcxmul(1,3)mul(1,2) line -> mul(8,7) enabled -> False --- After Iteration 4: No do and don't found result -> abcxmul(1,3)mul(1,2) break out of loop --- Result -> abcxmul(1,3)mul(1,2) ``` We process the result with the same `Multiply` function that we used in the first part after passing it through the `FindMulExpression` function that will return all the mul expressions in the result line string. ### Heads up with the input The actual input of the puzzle is I think multiple lines, so we need to preserve this state of the line in all the remaining lines. OR, we could be smarter and create a single large string and process that. Both are valid and would give the same results. I just didn’t like to add an overhead of keeping track of all the states and line, so I just concatenated all the lines and processed that single string. ## Conclusion This was a simple problem in essence but if you are not aware of regex you could go down a rabbit hole of writing your own parser or wired string manipulation (just like I did). That’s it form day 3, I will be doing more live stream solving over the weekend and may be the next week as well. Find the code for my AoC solutions here on GitHub. Till then, Happy Coding :) https://www.meetgor.com/posts/aoc-2024-day-2 Introduction So, this is day 2 of the Advent of Code 2024 in Golang, and we will be exploring my approach and solution for the same. The problem was not as easy Mon, 02 Dec 2024 00:00:00 UTC ## Introduction So, this is day 2 of the Advent of Code 2024 in Golang, and we will be exploring my approach and solution for the same. The problem was not as easy but was pretty simple after implemented and found correct. You can check out my solutions [here on GitHub](https://github.com/Mr-Destructive/advent_of_code/blob/main/2024/src/day02/main.go). ## Part 1 We have been given some lines called reports, and each report has a bunch of levels. and the requirement of the report is that it needs to be either strictly increasing or decreasing by a factor of at least 1 or at most 3. This means if the first two elements are increasing even by one, the other subsequent elements in that report should be increasing (by 1, 2, or 3) levels, and there cannot be any change (i.e. 0 change in two adjacent numbers, or two adjacent numbers cannot be same) ```plaintext 7 6 4 2 1 1 2 7 8 9 9 7 6 2 1 1 3 2 4 5 8 6 4 4 1 1 3 6 7 9 ``` * We first do the input parsing, it is pretty straightforward, we need to split it by ` ` to get individual reports, this will be a string so `”7 6 4 2 1"` , we want to get a slice of integers. * So we go ahead and split by spaces/whitespace `” “` to get the individual levels (numbers) and we need to convert them into integers. * Once we have individual strings of the report i.e. levels as `[“7”, “6”, “4”, “2”, “1”]` , we need to cast them to integers. * We iterate over each of them and cast them to integers and append to the list. * Once we have constructed the list, we append to the reports list which will be the array of arrays, i.e. each line is a report, and each report has many levels so slice of slice of integers. ```go func SplitLevels(lines []string) [][]int { reportLevels := [][]int{} for i, reportLine := range lines { reportLevels = append(reportLevels, []int{}) for _, levelStr := range strings.Split(reportLine, " ") { level, err := strconv.Atoi(levelStr) if err != nil { log.Fatal(err) } reportLevels[i] = append(reportLevels[i], level) } } return reportLevels } ``` Once we have constructed the reports and levels, we move ahead in actually analyzing the patterns within the levels in the individual reports. For that: * We first take individual reports, calculate the difference between the first two elements, and remember to use absolute difference carefully here. * We need to maintain a flag which indicates whether the levels in the report are increasing or decreasing, which can be determined with the first two elements. That is if the first two elements are increasing, the subsequent levels should also be increasing and if they are decreasing then all the levels should be decreasing as well * We first have a guard check, if the difference between them is 0 or greater than 3 or less than -3 which is the condition of the levels to be safe. If that is the case then we return false that is the report is not safe. * We now iterate on the report after the first two elements, we then compute the difference between the next two levels, if the flag is increasing is true and the current difference is less than or equal to 0 or it exceeding 3 we also mark it as false * The other condition is that if the flag is is decreasing, which means the first two elements were having a negative difference, so we check if the current difference is greater than or equal to 0 or it is less than -3, if that is the case we mark that as false * After computing the difference for all the levels, if we come out of the loop, we return true as we didn’t see any discrepancy in the levels. ```go func IsSafe(report []int) (bool) { prevDiff := report[1] - report[0] isIcreasing := prevDiff > 0 if prevDiff == 0 || prevDiff > 3 || prevDiff < -3 { return false } for i := 2; i < len(report); i++ { currDiff := report[i] - report[i-1] if isIcreasing { if currDiff <= 0 || currDiff > 3 { return false } } else { if currDiff >= 0 || currDiff < -3 { return false } } } return true } ``` ## Part 2 For part two, we need to do a few things, we need to compute if the report is safe or not, and if that is unsafe, we can almost remove one element from the report to make it safe. For that the approach is: * Get the index where we first saw the discrepancy in the levels * Check by removing that element from the report, if that makes the report safe, then return true i.e. we found the safe report * If we still find the report unsafe, remove the element before the index where the discrepancy was found, if now we find it safe after removing that element, then mark it safe * If still we find the report unsafe, then remove the element after the index where we originally found the discrepancy, if the report becomes safe, we mark that report safe * Else we mark the report unsafe, as we cannot find only the element removable that makes the report safe. ```go func RemoveAndCheck(report []int, index int) bool { if index > len(report)-1 || index < 0 { return false } reportNew := append([]int{}, report[:index]...) reportNew = append(reportNew, report[index+1:]...) safe, _ := IsSafe(reportNew) fmt.Println(safe, report) return safe } func RemoveLevels(report []int) bool { safe, unsafeIndex := IsSafe(report) if safe { return true } else { if RemoveAndCheck(report, unsafeIndex) { return true } if RemoveAndCheck(report, unsafeIndex-1) { return true } if RemoveAndCheck(report, unsafeIndex+1) { return true } return false } } ``` You can check out my solutions [here on GitHub](https://github.com/Mr-Destructive/advent_of_code/blob/main/2024/src/day02/main.go). ## Conclusion So that was it, a pretty simple problem for day 2 of the advent of code 2024 in Golang. Hope you enjoyed this walkthrough of the day one puzzle in the Advent of Code 2024 in Golang. Let me know if you have any other interesting solutions, or if you have anything to share about this, any feedback, questions, or suggestions are welcome. Thank you for reading, and I will see you tomorrow for day 3 Happy Coding :) https://www.meetgor.com/posts/aoc-2024-day-1 Introduction Hello everyone, it’s that time of the year, Advent of Code, I will be solving this year as well with Golang. In previous years I have been doing Sun, 01 Dec 2024 00:00:00 UTC ## Introduction Hello everyone, it’s that time of the year, Advent of Code, I will be solving this year as well with Golang. In previous years I have been doing Advent of Code but was not able to keep up with the pace and left it midway (not even halfway). This year however I am determined and want to solve all the problems as much as I can. Let’s dive into the first day which should be and is pretty simple and straightforward. A simple list and map creation and traversal and basic math operations. I also live-streamed the solution, you can check it out the [stream on YouTube](https://www.youtube.com/live/3K02tEEBgto?si=ojS5rsh5nGpk3U-B) And also a [shorter video](https://youtu.be/4U97gLyz0Ss?si=SvINHaGz-mow_q3O) on the approach and solution in Golang. <iframe width="560" height="315" src="https://www.youtube.com/embed/4U97gLyz0Ss" frameborder="0" allowfullscreen></iframe> Or you stick here and continue reading. Thank you You can check out my solutions [here on GitHub](https://github.com/Mr-Destructive/advent_of_code/blob/main/2024/src/day01/main.go). ## Part 1 [Advent of code, 2024, day 1](https://adventofcode.com/2024/day/1) We are given two lists here, the first part aims to find the absolute difference (distance) between each element sorted from smallest to largest. So, in essence, we take the two lists, sort them and one by one, for each corresponding element paired up, we take the absolute difference and sum that difference up for all the numbers in the list. ```plaintext 3 4 4 3 2 5 1 3 3 9 3 3 ``` So, first, we need to split the input into different lists: 1. We first range over all the lines, initialize two empty lists of integers 2. Then we split the line with the space as the separator, so this gives us the slice of strings as `[“3”, “4”]` 3. But we need to elements as integers, so take the first number and convert it to integer, similarly we do it for the second number. 4. Then once we have both numbers, we append them to the corresponding lists, the first number goes to the first list, and the second is appended to the second list. 5. Then we return the two lists NOTE: You cannot take the difference of those two numbers here itself, since we need to find the smallest number and sort the numbers in each list, so we need to get the lists populated first. ```go func SplitLists(lines []string) ([]int, []int) { listOne := []int{} listTwo := []int{} for _, line := range lines { // |3 4 // ["3","4"] slice of string ([]string) // 3 // 4 // [3,4] slice of int ([]int) numbers := strings.Split(line, " ") numOne, err := strconv.Atoi(numbers[0]) if err != nil { log.Fatal(err) } numTwo, err := strconv.Atoi(numbers[1]) if err != nil { log.Fatal(err) } listOne = append(listOne, numOne) listTwo = append(listTwo, numTwo) } return listOne, listTwo } ``` In the above code, we have initialized two empty slices of strings, we take the parameter lines which is a slice of string, indicating a line-by-line string representation of the input. I have parsed the input with these helper functions. The ReadFileBytes and ReadFileLines, one the bytes, the other gives the line-by-line string which gives a slice of strings. So once we have the lines, we iterate over each line and split the lines on space to get the two numbers. So, the line “`3 4`“ will be split into `[“3”, “4”]` . Now we get the first element and convert it into an integer as we need to sort and take the difference later. So, by accessing the first and second elements in the split line as `numbers[0]` and `numbers[1]` and converting the type to integer, [strconv.Atoi](https://pkg.go.dev/strconv#Atoi) function, which takes in a string and gives back an integer or an error. Now, we have two numbers as integers, we append the first element to the first element as `listOne = append(listOne, numOne)` and `listTwo = append(listTwo, numTwo)` So, we append one by one as we iterate over the input through all the lines, so at the end of this function, we will have two lists of integers. ``` [3 4 2 1 3 3] [4 3 5 3 9 3] ``` Then, once we have the slices of integers, we sort those lists. Then we range over the lists one by one element, since both the lists are of the same size, we can reference one by the index of the other. Then for each difference of the two integers (one from the first list and the other from the second list), we cast it to a `float64` and pass it to the [math.Abs](https://pkg.go.dev/math#Abs) function, which is annoying as Golang doesn’t have an absolute function for intgers. We cast the integer to float for parsing it to the Abs method and cast the returned float64 value back to int. Kind of wired but fine. We keep adding the absolute differences for each paired difference of the elements in the two lists. At the end, we will have a final score which is the score for part one. ```go func PartOne(lines []string) int { listOne, listTwo := SplitLists(lines) sort.Ints(listOne) sort.Ints(listTwo) totalScore := 0 for i := range listOne { totalScore += int(math.Abs(float64(listOne[i] - listTwo[i])) } return totalScore } ``` ## Part 2 For part two, we need to take all the numbers in the first list count the number of times that number has occurred in the second list, and take a product of them and add it up for all the numbers. So in the example: ```plaintext 3 4 4 3 2 5 1 3 3 9 3 3 ``` The numbers in the first list are \[3,4,2,1,3,3\] We have to count the occurrences of each of them in the second list So, in the second list \[4,3,5,3,9,3\], the number `3` occurs `3` times, so we do `3×3` which is `9` and then, do the same for `4` which occurs only once in the second list so, we get `4`, then `2` occurs `0` times, so we get `0` We get → `(3×3) + (4×1) + (2×0) + (1×0) + (3×3) + (3×3)` The first number is the element in the first list and the second number is the occurrence of that number in the second list. which comes out to be `9+4+0+0+9+9` , so the answer is `31` for the example. Once it is clear, what we have to do, we simply have to iterate over the second list and create a map of the frequency/occurances/number of times it appears in that list. ### Solution So, we will have to modify the `SplitLists` functions a bit, we need to split and also map the second list with the key as the number itself and the value as its count in the second list. Just that change, we create an additional return value with an empty map of integers with integers. The mapTwo variable will be a map that will have a key as the number in the second list and its value as the number of times it is present in that list. ```go func SplitListsAndMap(lines []string) ([]int, map[int]int) { listOne := []int{} listTwoCounts := make(map[int]int) for _, line := range lines { numbers := strings.Split(line, " ") numOne, err := strconv.Atoi(numbers[0]) if err != nil { log.Fatal(err) } numTwo, err := strconv.Atoi(numbers[1]) if err != nil { log.Fatal(err) } listOne = append(listOne, numOne) listTwoCounts[numTwo] += 1 } return listOne, listTwoCounts } ``` So, as we iterate over each line, we parse the string number into an integer and increment its count in the map. ``` [3 4 2 1 3 3] map[3:3 4:1 5:1 9:1] ``` In the actual calculation of the score, we need to iterate over the elements of the first list and multiply the number with its count in the second list as we now have the map of it. We multiply those and add them up for each line, which becomes the final score. ```go func PartTwo(lines []string) int { similarityScore := 0 listOne, mapTwo := SplitListsAndMap(lines) for _, numOne := range listOne { score := numOne * mapTwo[numOne] similarityScore += score } return similarityScore } ``` So, that is how we got the final score for part two. You can check out my solutions [here on GitHub](https://github.com/Mr-Destructive/advent_of_code/blob/main/2024/src/day01/main.go). ## Conclusion So that was it, a pretty simple problem for day 1 of the advent of code 2024 in Golang. Hope you enjoyed this walkthrough of the day one puzzle in the Advent of Code 2024 in Golang. Thank you for reading, and I will see you tomorrow for day 2 Happy Coding :) https://www.meetgor.com/posts/turso-libsql-embedded-replicas-golang Introduction Welcome to the Let's Go with Turso series. In this series, we will learn how to interact with LibSQL databases with Golang. In the past article of Thu, 31 Oct 2024 00:00:00 UTC ## Introduction Welcome to the Let's Go with Turso series. In this series, we will learn how to interact with LibSQL databases with Golang. In the past article of the series, we explored how to connect remote/local LibSQL database in golang. With this section, we will specifally dive into understanding how to create, connect, and query local embedded replicas of LibSQL database hosted on Turso with a Golang application. If you want to check out the YouTube video, check this out: [LibSQL Embedded Replicas on Turso in Golang](https://www.youtube.com/watch?v=BitxB40rdVw) <iframe width="560" height="315" src="https://www.youtube.com/embed/vBrvX0X0phw" frameborder="0" allowfullscreen></iframe> ## LibSQL Embedded Replicas on Turso in Golang ### Embedded Replicas The embedded replica is a native feature of the libSQL database. With embedded replicas, we can have faster writes as with the local database and global read access from the remote database. Embedded replica is like creating a copy of a primary remote database and using it for performing any operations from the applications as a local database and then it has a mechanism or standard to sync up with the primary remote database. The primary remote database serves as a single source of truth, however we can use the database locally as well. This however should be done carefully to avoid data corruption or stale data. To cope up with this stale or corruption of data, the periodic syncing can be used. Embedded replicas have a 3 fold process - Create a copy of the primary remote database - Perform any operations on the local database - Sync up with the primary remote database  There are a few things to remember here: - The local database can read it's newly written data, however other local replica databases can only view those changes once the sync has happened and the primary database has been updated from the local copy. - These would require to sync the local with the primary database first and then the other local database also needs to sync with the primary database. You can read more about it [here](https://docs.turso.tech/features/embedded-replicas/introduction) on the Turso documentation. Let's get started, What we are going to do is create a step by step procedure to understand how to work with embedded replicas of LibSQL database. 1. Perform the operations on the local LibSQL database 2. Create a Embedded Replica and sync up with the primary remote database 3. Fetch data from the primary remote database  In this way, we can understand how to operate the Embedded Replicas as a whole from locally as well as remotely ## Initializing a Golang project Let's start with initializing a Golang project. ```bash # go mod init <git-provider-domain>/<username>/<project-name> # Example go mod init github.com/mr-destructive/lets-go-with-turso ``` This will initialize the project in the current directory, creating a `go.mod` file with the specification of the Golang version and the packages that we will install and use in this module. ## Installing go-libsql package We will need to install the [go-libsql](https://github.com/tursodatabase/go-libsql) package, this is the driver for LibSQL for Golang. To install simply use the `go get` command to be installed as the dependency for the project ```bash go get github.com/tursodatabase/go-libsql ``` Once this is done, we can create a local LibSQL database. ## Creating a local LibSQL database Let's create a local LibSQL database. With the `go-libsql` package, we can connect to the local database. This will be done using the `libsql` driver. There is really no much difference than the normal SQLite database driver, this works just like SQLite, the only difference being the ability to connect to the remote database. ```go package main import ( "database/sql" "fmt" "os" _ "github.com/tursodatabase/go-libsql" ) func main() { dbName := "file:./local.db" db, err := sql.Open("libsql", dbName) if err != nil { fmt.Fprintf(os.Stderr, "failed to open db %s", err) os.Exit(1) } defer db.Close() } ``` The above code will simply connect to the local LibSQL database which resides as the `local.db` file. Now, to demonstrate that it is an actual sqlite-like database, we can execute queries on the connected database. ```go package main import ( "database/sql" "fmt" "os" _ "github.com/tursodatabase/go-libsql" ) func main() { dbName := "file:./local.db" db, err := sql.Open("libsql", dbName) if err != nil { fmt.Fprintf(os.Stderr, "failed to open db %s", err) os.Exit(1) } defer db.Close() rows, err := db.Query("SELECT ABS(RANDOM()%5) FROM generate_series(1,10)") if err != nil { fmt.Fprintf(os.Stderr, "failed to query %s", err) os.Exit(1) } defer rows.Close() for rows.Next() { var i int if err := rows.Scan(&i); err != nil { fmt.Fprintf(os.Stderr, "failed to scan %s", err) os.Exit(1) } fmt.Println(i) } } ``` Output: ``` $ go run main.go 3 0 4 4 2 1 2 3 4 1 $ go run main.go 0 2 1 2 3 2 2 1 0 2 ``` Just a simple `SELECT ABS(RANDOM()%5) FROM generate_series(1,10)` query will be executed on the connected database. This query will basically genrate a random number between `-4` and `4` and take the absolute value i.e. now between 0 and 4, this will genrate 10 such numbers. Now, we can move into the actual embedded replica creation of the primary remote database and use it as a local database to perform operations. ## Creating an Embedded Replica of Turso's LibSQL database We need a few things to specify before we create the embedded replica, first being the primary remote database, that typically is a libsql database hosted on turso or self hosted. We also need to provide the local database path, where the local-replica will be stored or we can say the copy of the primary libsql database will be created. We will be using the [LibSQL.NewEmbeddedReplicaConnector](https://pkg.go.dev/github.com/levydsa/libsql-go#NewEmbeddedReplicaConnector) to create a local replica of a libsql database. The function takes in two paramters, the first paramter being the local database filename path to create the copy into, and the second paramter being the primary database URL. The function returns a connector object or an error if any. The connector object is then further used with [OpenDB](https://pkg.go.dev/database/sql#OpenDB) function to create a database connection. The `OpenDB` function returns a reference of database connections which we'll use to connect and perform operations on the database. The same `connector` object could be used to sync with the primary database after performing operations on the local database with the [Sync](https://pkg.go.dev/github.com/levydsa/libsql-go#Connector.Sync) method. This will pull or push the changes from the local database to the primary database. We can configure the syncing mechanism while creating the embedded replica with the additional parameters to the `NewEmbeddedReplicaConnector` function. There are [Options](https://pkg.go.dev/github.com/levydsa/libsql-go#Option) to include for the paramters that could be passed like: - `WithAuthToken(string)`: This will be used to authenticate with the primary database. - `WithSyncInterval(time.Duration)`: This will be used to specify the interval of syncing between the local and primary database. - `WithEncrytion(string)`: This will be used to encrypt the local database. - `WithReadYourWrites(bool)`: This will be used to specify if the local database can read the newly written changes or not. So, let's create a exmaple to create a embedded replica, make some changes by creating tables and then syncing the local with primary, finally appending some data to the local and reading those. #### Create the Embedded Replica We first need to create a copy of the primary database, as said, we will configure the 2 paramters that we need to create the embedded replica with `NewEmbeddedReplicaConnector`. Then once we have the connector object, we open up a database connection, at that point we can further run queries on the local replica that was just created from the primary remote LibSQL database. ```go package main import ( "database/sql" "fmt" "os" "path/filepath" "github.com/tursodatabase/go-libsql" ) func main() { dbName := "local.db" // this is not required, but can be used to create a temporary directory and then delete it later dir, err := os.MkdirTemp("", "libsql-*") if err != nil { fmt.Println("Error creating temporary directory:", err) os.Exit(1) } defer os.RemoveAll(dir) // first paramter required for creating NewEmbeddedReplicaConnector dbPath := filepath.Join(dir, dbName) fmt.Println(dbPath) // second paramter required for creating NewEmbeddedReplicaConnector dbURL := os.Getenv("TURSO_DATABASE_URL") dbAuthToken := os.Getenv("TURSO_AUTH_TOKEN") connector, err := libsql.NewEmbeddedReplicaConnector(dbPath, dbURL, libsql.WithAuthToken(dbAuthToken)) fmt.Println(connector) if err != nil { fmt.Fprintf(os.Stderr, "failed to open db %s", err) os.Exit(1) } defer connector.Close() // open a database connection from the connector object db := sql.OpenDB(connector) fmt.Println("Connected to database") if err != nil { fmt.Fprintf(os.Stderr, "failed to open db %s", err) os.Exit(1) } defer db.Close() } ``` In the above code, we first create a temporary directory with the help of [MkdirTemp](https://pkg.go.dev/os#MkdirTemp), this is not required, but would be easier for cleanup later. We then the path for the local database to be created. The combined path string `dbPath` will serve as the first paramter to the `NewEmbeddedReplicaConnector`. Then we have taken the `dbURL` and the `dbAuthToken` from the environment variables `TURSO_DATABASE_URL` and `TURSO_AUTH_TOKEN` respectively. The `dbURL` will serve as the second paramter for the `NewEmbeddedReplicaConnector` that is the URL of the primary remote LibSQL database. The function `NewEmbeddedReplicaConnector` will return the `Connector` object if successfull in creation of the replica, else return `err` if it fails. The connector object needs to be closed at the end of the program, so we use the `defer connector.Close()` that will close the connection to the primary database at the end of the program. The `sql.OpenDB` is used to create the connection with the local database that will be created from the `connector` object. Finally we also need to close the local database connection at the end of the program. Further, we will try to query the local replica and create tables and append data to it. ### Adding data to teh local replica Once we have the `db` connection to the local database, we can noramlly query the database as we did in the previous example, of querying the local LibSQL database. Let's start by creating a table `posts` to the local replica, this will basically create the schema in the local database. ```go .... createPostTableQuery := `CREATE TABLE IF NOT EXISTS posts( id INTEGER PRIMARY KEY, title VARCHAR(100), description VARCHAR(255), content TEXT );` _, err = db.Exec(createPostTableQuery) if err != nil { fmt.Fprintf(os.Stderr, "failed to create table %s", err) os.Exit(1) } ``` The `createPostTableQuery` will be the `SQL` to create the table `posts` if it doesn't already exist in the database (local replica). Then with the help of [db.Exec](https://pkg.go.dev/database/sql#DB.Exec) function, we can execute the query and return back the rows if it had created any. In this case it won't as we have just added a table. Then, we can either sync the database to the primary, but let's populate the table `posts` with some data before syncing with the primary db. ```go createPostQuery := `INSERT INTO posts(title, description, content) VALUES(?, ?, ?)` _, err = db.Exec(createPostQuery, "test title", "test description", "test content") if err != nil { fmt.Fprintf(os.Stderr, "failed to insert %s", err) os.Exit(1) } ``` We have created the `createPostQuery` similarly to insert into the `posts` table in the local replica. The values are added with the placeholders in the `Exec` function as positional parameters. Once we have executed the query, this will populate the `posts` table in the lcoal replica. We can now finally sync with the primary remote LibSQL database to make sure that the primary database also has these migrations applied. ### Syncing the local replica Remember, `connector` is for primary database and `db` is for the local replica. So, we will sync the remote database from the replica that was created with the `connector.Sync` ```go _, err = connector.Sync() if err != nil { fmt.Fprintf(os.Stderr, "failed to sync %s", err) os.Exit(1) } fmt.Printf("Successfully synced %s db ", dbPath) rows, err := db.Query("SELECT * FROM posts") if err != nil { fmt.Fprintf(os.Stderr, "failed to query %s", err) os.Exit(1) } defer rows.Close() for rows.Next() { var id int var title string var description string var content string if err := rows.Scan(&id, &title, &description, &content); err != nil { fmt.Fprintf(os.Stderr, "failed to scan %s", err) os.Exit(1) } fmt.Println(id, title, description, content) } ``` Output: ```bash $ go run main.go /tmp/libsql-349052144/local.db &{0x2eec9d0 <nil> <nil>} Connected to database Successfully synced /tmp/libsql-349052144/local.db db 1 test title test description test content ``` Once we have synced the local replica, we can now query the database i.e. the local replica, with the changes, also note that this could also be done without syncing the database, but the primary database won't have the applied changes. We finally Query the local replica with the query `SELECT * FROM posts` and print out the results. This has the 1 record in the `posts` table that we inserted. So, that's how we basically create a local replica from a remote LibSQL database hosted on Turso. We first create the path for the local database to be copied, then provide the primary database URL and credentials, then request a copy of the primary database, then we perform any mutation or operations on the local copy and finally sync up with the remote primary database to persist the data from that replica (acting like a session of database operation). That wraps the article for now. For all the code related to this article, you can check out the [Let's Go with Turso](https://github.com/mr-destructive/lets-go-with-turso) GitHub repo for all the examples and additional examples for using LibSQL with Golang. ## Conclusion So, that is a wrap for this part of the series, we have explored how to create a local embedded replica from a remote LibSQL database hosted on Turso with Golang. In the next part of the series, we will explore how to setup a local LibSQL database server and then connect it with Golang. Thank you for reading this post, If you have any questions, feedback, and suggestions, feel free to drop them in the comments. Happy Coding :) https://www.meetgor.com/posts/turso-libsql-db-golang Introduction Welcome to the new series in Golang, Let's Go with Turso. In this series, we will learn how to interact with LibSQL databases with Golang. We will Mon, 30 Sep 2024 00:00:00 UTC ## Introduction Welcome to the new series in Golang, Let's Go with Turso. In this series, we will learn how to interact with LibSQL databases with Golang. We will connect with a remote/local LibSQL database, create Embedded replicas, set up a local LibSQL database, and so much more as we explore and find out more features of LibSQL. ## Connect a LibSQL database in a Golang application In this post, we will learn how to connect and query a LibSQL database hosted on Turso/Cloud in a Golang Application using libsql-client package. We will go from setting up golang project, installing turso-cli, creating a database on turso with the cli, connecting to the database with shell, and golang and finally, we can query the database using Golang. If you want to check out the YouTube video, check this out: [Conenct LibSQL Database hosted on Turso with Golang](https://www.youtube.com/watch?v=vBrvX0X0phw) <iframe width="560" height="315" src="https://www.youtube.com/embed/vBrvX0X0phw" frameborder="0" allowfullscreen></iframe> ### Initializing a Golang project Let's start with initializing a Golang project. ```bash # go mod init <git-provider-domain>/<username>/<project-name> # Example go mod init github.com/mr-destructive/lets-go-with-turso ``` This will initialize the project in the current directory, creating a `go.mod` file with the specification of the Golang version and the packages that we will install and use in this module. ### Installing Turso CLI ```bash # Linux/Windows curl -sSfL https://get.tur.so/install.sh | bash curl -sSfL https://get.tur.so/install.sh | bash # macOS brew install tursodatabase/tap/turso ``` This will install the Turso CLI. To verify that Turso CLI is installed properly, you can run the version command to check the setup. ``` turso --version ``` Once it is installed, we can now log in into Turso platform, simply by running the `auth signup` or `auth login` to Register or Log-in. ``` turso auth signup # OR turso auth login ``` This will redirect to the browser for the OAuth flow, once signed up and logged in, this will allow to interact with the Turso platform with the CLI that we downloaded. To make sure we are logged in as the correct user, we can run the `auth whoami` command to get the currently logged-in user. ``` turso auth whoami ``` This will print the username if you are logged-in. If everything seems correct, we can move ahead with the database creation step. ### Creating a Remote LibSQL Database on Turso To create a LibSQL database hosted on Turso, we will use the `turso db create` command. ``` turso db create # OR turso db create <name> ``` This will create a database with the specified name, even if you don't provide the name, it will give out a random friendly two-word name to your database. It will create a database on the nearest location available from your location. This command will output the following: ``` Created database <db-name> at group default in 1.99s. Start an interactive SQL shell with the following: turso db shell <db-name> To see information about the database, including a connection URL, run: turso db show <db-name> To get an authentication token for the database, run: turso db tokens create <db-name> ``` The next step, it shows to start an interactive shell, to see information about the database, and to generate an authentication token for the database. We will move to the next part, which would be to create an authentication token for accessing the database from an external application. ### Generating and Storing Authentication Token for LibSQL Database After we executed the `db create` command, and it created the database on the Turso cloud, there was a command hint for creating a `token` with the command `db tokens create` So, this command will create a JWT authentication token, that will be used to connect and read/write to the database. ```bash turso db tokens create <db-name> # OR turso db tokens create <db-name> --read-only # OR turso db tokens create <db-name> --expiration 30d ``` We can use the simple `db tokens create <db-name>` to create an authentication token for the database with (read + write access). You can copy that returned token into a environment variable, or wherever your application can read that token. This could be stored in the environment variable as follows: ```bash export TURSO_AUTH_TOKEN="<token>" ``` To make a `read-only` token, we can use the flag `--read-only`. This will be handy, if you only have a database as a local replica, and the only purpose of the database is for querying data. This will prevent any write operation on the database. We can also use the `--expiration` flag that will be used to set the duration of the token. By default the value for expiry is `never`, but that could be a little too risky if you are making a serious application. You can either set it to `7d` which will make the token expire after seven days. Now, we can get the remote database URL and connect to the database. The URL could be obtained by running the command `db show <db-name>` ``` turso db show <db-name> ``` This will output the following: ```bash Name: <db-name> URL: libsql://<db-name>-<username>.turso.io ID: <db-id> Group: default Version: 0.24.22 Locations: bom Size: 4.1 kB Archived: No Bytes Synced: 0 B Is Schema: No Database Instances: NAME TYPE LOCATION bom primary bom ``` The above output shows the meta-information of the database. This also has the URL hosted on Turso. We can construct the URL using the name of the database and your username as `libsql://<db-name>-<username>.turso.io`, you can set this in an environment variable or in the configuration wherever you can access it from the application. To set the URL of the database in your application, you can use the `TURSO_DB_URL` environment variable. ```bash export TURSO_DATABASE_URL="libsql://<db-name>-<username>.turso.io" ``` So, we have a remote database URL, and the access token configured, these are the two pieces that we will need to connect, read and write to the libsql database. ### Installing LibSQL Client for Golang So, we can install the [libsql-client-go](https://pkg.go.dev/github.com/tursodatabase/libsql-client-go/libsql) package for Golang which will be used as an SDK in Golang to interact with a remote LibSQL database. ```bash go get github.com/tursodatabase/libsql-client-go/libsql ``` This will install the package `libsql` into the golang module. Now, we can use this in our golang application. ### Populating the LibSQL Database Moving ahead, we have a database, but it doesn't have data! So let's create some tables and insert some rows. We can use the `db shell` command to open an interactive SQL shell on a remote LibSQL database. ```bash turso db shell libsql://<db-name>-<username>.turso.io ``` This will be a default `sqlite3` like a shell, where we can execute SQL commands, like `.schema`, `.mode`, `.tables`, etc. ```bash .dump Render database content as SQL .help List of all available commands. .indexes List indexes in a table or database .mode Set output mode .quit Exit this program .read Execute commands from a file .schema Show table schemas. .tables List all existing tables in the database. ``` And definitely, we can use the normal SQL queries, to read, write and delete data from the database. #### Creating a Table First, let's create a simple table, called `posts` with columns like `id`, `title` as a `VARCHAR(100)`, `description` as a `VARCHAR(255)`, and `content` as `TEXT` which won't be `NULL`. ```sql CREATE TABLE posts ( id INTEGER PRIMARY KEY, title VARCHAR(100), description VARCHAR(255), content TEXT NOT NULL ); ``` This will create a table `posts` on the LibSQL database, yes this will mutate the primary LibSQL database which is hosted on Turso. #### Inserting Rows Now, since we have the `posts` table, we will insert some rows into the table. ```bash INSERT INTO posts (title, description, content) VALUES ('test title', 'test description', 'test content'); ``` Now, we have some rows populated in the `posts` table. We can add more tables, and rows into the database, as usual, but this is just an example, so I'll keep it short. ### Connecting to the LibSQL Database Now, we have something to query from a database, after we connect to the database via the Golang application program. First, we will grab two pieces to connect to the database. ```bash export TURSO_DATABASE_URL="libsql://<db-name>-<username>.turso.io" export TURSO_AUTH_TOKEN="<token>" ``` Now, let's start with the Golang program code. ```go package main import ( "fmt" "os" ) func main() { dbURL := os.Getenv("TURSO_DATABASE_URL") dbToken := os.Getenv("TURSO_AUTH_TOKEN") dbUrl := fmt.Sprintf("%s?authToken=%s", dbURL, dbToken) } ``` This will be the basic config to grab the database URL and the authentication token, then we can construct the `dbURL` along with `dbToken` to construct the complete dbURL which will allow to access the database. Moving ahead, we will import `database/sql` package that will be used to open the database connection and `github.com/tursodatabase/libsql-client-go/libsql` to connect to the remote LibSQL database. ```go package main import ( "database/sql" "fmt" "os" _ "github.com/tursodatabase/libsql-client-go/libsql" ) func main() { dbURL := os.Getenv("TURSO_DATABASE_URL") dbToken := os.Getenv("TURSO_AUTH_TOKEN") dbUrl := fmt.Sprintf("%s?authToken=%s", dbURL, dbToken) db, err := sql.Open("libsql", dbUrl) if err != nil { fmt.Fprintf(os.Stderr, "failed to open db %s: %s", dbUrl, err) os.Exit(1) } defer db.Close() } ``` The `sql.Open` function will open the connection to the database, this will return a `sql.DB` object. The driver selected is `libsql` and the `dbURL` is the entire URL along with the authentication token. ```go type Post struct { Id int Title string Description string Content string } rows, err := db.Query("SELECT * FROM posts") if err != nil { fmt.Fprintf(os.Stderr, "failed to query: %s", err) os.Exit(1) } for rows.Next() { var post Post if err := rows.Scan(&post.Id, &post.Title, &post.Description, &post.Content); err != nil { fmt.Fprintf(os.Stderr, "failed to scan: %s", err) os.Exit(1) } fmt.Println(post) } defer rows.Close() ``` Now, let's query some data from the database. We can construct the `Post` struct that will be used to grab the required fields like `Id`, `Title`, `Description`, and `Content` from the `posts` table in the database. Then, we will use the `db.Query` function to query the database. This function takes in a query and returns a `sql.Rows` object. We will iterate over all the `rows` returned from the database, with the `rows.Next()` that will fetch each row. Then we can `row.Scan` the row object with the appropriate and respective fields returned in the row. In this case, the `Id`, `Title`, `Description`, and the `Content` is fetched and stored into the `post` fields. We have fetched the rows and we can do operations on them as required, this was just a basic example. So the entire code can be found below. ```go package main import ( "database/sql" "fmt" "os" _ "github.com/tursodatabase/libsql-client-go/libsql" ) type Post struct { Id int Title string Description string Content string } func main() { dbURL := os.Getenv("TURSO_DATABASE_URL") dbToken := os.Getenv("TURSO_AUTH_TOKEN") dbUrl := fmt.Sprintf("%s?authToken=%s", dbURL, dbToken) db, err := sql.Open("libsql", dbUrl) if err != nil { fmt.Fprintf(os.Stderr, "failed to open db %s: %s", dbUrl, err) os.Exit(1) } defer db.Close() rows, err := db.Query("SELECT * FROM posts") if err != nil { fmt.Fprintf(os.Stderr, "failed to query: %s", err) os.Exit(1) } for rows.Next() { var post Post if err := rows.Scan(&post.Id, &post.Title, &post.Description, &post.Content); err != nil { fmt.Fprintf(os.Stderr, "failed to scan: %s", err) os.Exit(1) } fmt.Println(post) } defer rows.Close() } ``` The output of the above code will result in all the rows present in the post table of the LibSQL database. ```bash $ go run remote.go {1 test title test description test content} {2 test title test description test content} {3 sample post libsql tutorial create db, connect, create tables, insert rows, sync} {4 test title test description test content} ``` I have added a few more rows to the post table, as you can see we have successfully connected, inserted, and read from the post table in the LibSQL database hosted on Turso. For all the code related to this article, you can check out the [Let's Go with Turso](https://github.com/mr-destructive/lets-go-with-turso) GitHub repo for all the examples and additional examples for using LibSQL with Golang. ## Conclusion So, that is a wrap for this part of the series, we have explored how to connect a remote LibSQL database hosted on Turso with Golang. In the next part of the series, we will explore how to create embedded replicas on Turso's LibSQL database in Golang. Thank you for reading this post, If you have any questions, feedback, and suggestions, feel free to drop them in the comments. Happy Coding :) https://www.meetgor.com/posts/golang-safely-using-maps Introduction This week, I was working on one of the API wrapper packages for golang, and that dealt with sending post requests with URL encoded values, setting Sat, 31 Aug 2024 00:00:00 UTC ## Introduction This week, I was working on one of the API wrapper packages for golang, and that dealt with sending post requests with URL encoded values, setting cookies, and all the fun stuff. However, while I was constructing the body, I was using [url.Value](https://pkg.go.dev/net/url#Values) type to construct the body, and use that to add and set key-value pairs. However, I was getting a wired `nil` pointer reference error in some of the parts, I thought it was because of some of the variables I set manually. However, by debugging closer, I found out a common pitfall or bad practice of just declaring a type but initializing it and that causing `nil` reference errors. In this post, I will cover, what are maps, how to create maps, and especially how to properly declare and initialize them. Create a proper distinction between the declaration and initialization of maps or any similar data type in golang. ## What is a map in Golang? A [map](https://go.dev/src/runtime/map.go) or a hashmap in golang is a basic data type that allows us to store key-value pairs. Under the hood, it is a header map-like data structure that holds buckets, which are basically pointers to bucket arrays (contiguous memory). It has hash codes that store the actual key-value pairs, and pointers to new buckets if the current overflows with the number of keys. This is a really smart data structure that provides almost constant time access. ## How to create maps in Golang To create a simple map in golang, you can take an example of a letter frequency counter using a map of string and integer. The map will store the letters as keys and their frequency as values. ```go package main import "fmt" func main() { words := "hello how are you" letters := map[string]int{} for _, word := range words { wordCount[word]++ } fmt.Println("Word counts:") for word, count := range wordCount { fmt.Printf("%s: %d ", word, count) } } ``` ```bash $ go run main.go Word counts: e: 2 : 3 w: 1 r: 1 y: 1 u: 1 h: 2 l: 2 o: 3 a: 1 ``` So, by initializing the map as `map[string]int{}` you will get an empty map. This can be then used to populate the keys and values, we iterate over the string, and for each character (rune) we cast that byte of character into the string and increment the value, the zero value for int is `0`, so by default if the key is not present, it will be zero, it is a bit of double-edged swords though, we never know the key is present in the map with the value `0` or the key is not present but the default value is `0`. For that, you need to check if the `key` exists in the map or not. For further details, you can check out my [Golang Maps](https://www.meetgor.com/golang-maps/) post in detail. ## Difference between declaration and initialization There is a difference in declaring and initializing any variable in a programming language and has to do a lot more with the implementation of the underlying type. In the case of primary data types like `int`, `string`, `float`, etc. there are default/zero values, so that is the same as the declaration and initialization of the variables. However, in the case of maps and slices, the declaration is just making sure the variable is available to the scope of the program, however for initialization is setting it to its default/zero value or the actual value that should be assigned. So, declaration simply makes the variable available within the scope of the program. For maps and slices, declaring a variable without initialization sets it to `nil`, meaning it points to no allocated memory and cannot be used directly. Whereas the `initialization` allocates memory and sets the variable to a usable state. For maps and slices, you need to explicitly initialize them using syntax like `myMap = make(map[keyType]valueType)` or `slice = []type{}`. Without this initialization, attempting to use the map or slice will lead to runtime errors, such as panics for accessing or modifying a nil map or slice. Let's looks at the values of a map when it is declared/initialized/not initialized. Imagine you're building a configuration manager that reads settings from a map. The map will be declared globally but initialized only when the configuration is loaded. 1. Declared but not initialized The below code demonstrates a map access that is not initialized. ```go package main import ( "fmt" "log" ) // Global map to store configuration settings var configSettings map[string]string // Declared but not initialized func main() { // Attempt to get a configuration setting before initializing the map serverPort := getConfigSetting("server_port") fmt.Printf("Server port: %s ", serverPort) } func getConfigSetting(key string) string { if configSettings == nil { log.Fatal("Configuration settings map is not initialized") } value, exists := configSettings[key] if !exists { return "Setting not found" } return value } ``` ```bash $ go run main.go Server port: Setting not found ``` 2. Declared and Initialized at the same time The below code demonstrates a map access that is initialized at the same time. ```go package main import ( "fmt" "log" ) // Global map to store configuration settings var configSettings = map[string]string{ "server_port": "8080", "database_url": "localhost:5432", } func main() { serverPort := getConfigSetting("server_port") fmt.Printf("Server port: %s ", serverPort) } func getConfigSetting(key string) string { value, exists := configSettings[key] if !exists { return "Setting not found" } return value } ``` ```bash $ go run main.go Server port: 8080 ``` 3. Declared and later initialized The below code demonstrates a map access that is initialized later. ```go package main import ( "fmt" "log" ) // Global map to store configuration settings var configSettings map[string]string // declared but not initialized func main() { // Initialize configuration settings initializeConfigSettings() // if the function is not called, the map will be nil // Get a configuration setting safely serverPort := getConfigSetting("server_port") fmt.Printf("Server port: %s ", serverPort) } func initializeConfigSettings() { if configSettings == nil { configSettings = make(map[string]string) // Properly initialize the map configSettings["server_port"] = "8080" configSettings["database_url"] = "localhost:5432" fmt.Println("Configuration settings initialized") } } func getConfigSetting(key string) string { if configSettings == nil { log.Fatal("Configuration settings map is not initialized") } value, exists := configSettings[key] if !exists { return "Setting not found" } return value } ``` ```bash $ go run main.go Configuration settings initialized Server port: 8080 ``` In the above code, we declared the global map `configSettings` but didn't initialize it at that point, until we wanted to access the map. We initialize the map in the main function, this main function could be other specific parts of the code, and the global variable `configSettings` a map from another part of the code, by initializing it in the required scope, we prevent it from causing nil pointer access errors. We only initialize the map if it is `nil` i.e. it has not been initialized elsewhere in the code. This prevents overriding the map/flushing out the config set from other parts of the scope. ## Pitfalls in access of un-initialized maps But since it deals with pointers, it comes with its own pitfalls like nil pointers access when the map is not initialized. Let's take a look at an example, a real case where this might happen. ```go package main import ( "fmt" "net/url" ) func main() { var vals url.Values vals.Add("foo", "bar") fmt.Println(vals) } ``` This will result in a runtime panic. ```nginx $ go run main.go panic: assignment to entry in nil map goroutine 1 [running]: net/url.Values.Add(...) /usr/local/go/src/net/url/url.go:902 main.main() /home/meet/code/playground/go/main.go:10 +0x2d exit status 2 ``` This is because the [url.Values](https://pkg.go.dev/net/url#Values) is a map of string and a list of string values. Since the underlying type is a map for `Values`, and in the example, we only have declared the variable `vals` with the type `url.Values`, it will point to a `nil` reference, hence the message on adding the value to the type. So, it is a good practice to use `make` while declaring or initializing a map data type. If you are not sure the underlying type is `map` then you could use `Type{}` to initialize an empty value of that type. ```go package main import ( "fmt" "net/url" ) func main() { vals := make(url.Values) // OR // vals := url.Values{} vals.Add("foo", "bar") fmt.Println(vals) } ``` ```bash $ go run urlvals.go map[foo:[bar]] foo=bar ``` It is also recommended by the [golang team](https://go.dev/blog/maps) to use the make function while initializing a map. So, either use `make` for maps, slices, and channels, or initialize the empty value variable with `Type{}`. Both of them work similarly, but the latter is more generally applicable to structs as well. ## Conclusion Understanding the difference between declaring and initializing maps in Golang is essential for any developer, not just in golang, but in general. As we've explored, simply declaring a map variable without initializing it can lead to runtime errors, such as panics when attempting to access or modify a nil map. Initializing a map ensures that it is properly allocated in memory and ready for use, thereby avoiding these pitfalls. By following best practices—such as using the make function or initializing with Type{}—you can prevent common issues related to uninitialized maps. Always ensure that maps and slices are explicitly initialized before use to safeguard against unexpected nil pointer dereferences Thank you for reading this post, If you have any questions, feedback, and suggestions, feel free to drop them in the comments. Happy Coding :) https://www.meetgor.com/posts/nginx-02-web-servers Introduction In the second part of our NGINX Survival Guide, we dive into the practical aspects of using NGINX to serve web applications. This section will guid Sun, 21 Jul 2024 00:00:00 UTC ## Introduction In the second part of our NGINX Survival Guide, we dive into the practical aspects of using NGINX to serve web applications. This section will guide you through the essential tasks of setting up a basic HTTP server, configuring NGINX to serve content from custom directories, and using it as a reverse proxy to forward requests to backend servers. NGINX is a versatile web server that can be used to serve applications in a variety of ways, from simple web servers to complex proxy configurations. NGINX can be used to serve static HTML content, proxy requests to a backend server, or load balance traffic across multiple servers. In this guide, we'll explore the different ways to use NGINX to serve applications, including setting up a simple HTTP server, serving content from custom directories, and using it to load balance traffic across multiple upstream servers. ## Simple HTTP Server NGINX serves as the default HTTP server on port 80 of your local machine if NGINX is properly installed and running on your system. If you head on to the localhost, you will see the default NGINX HTML page like the one below:  This is the default HTML page served by NGINX as per the configuration in the `/etc/nginx/nginx.conf` file. The default folder for NGINX to serve HTML content is located at `/usr/share/nginx/html/index.html` , If you change the contents of this file and restart NGINX, the http server will load the new HTML content. Let's first look, at how we can serve a simple http message within the configuration file in NGINX. ## Serving simple text We will try to write our simple HTTP server from scratch, so it would be nice to empty the existing `/etc/nginx/nginx.conf` file or use other ports to serve the content rather than the default `127.0.0.1:80` port. ```nginx http { server { listen 8000; return 200 "Hello, World! "; } } ``` The above config will serve the text `Hello, World!` when there is a request to the URL `127.0.0.1:8000` or `localhost:8000` You can change the port per your requirements and even add a `server_name` for your domain name. ```bash $ curl http://127.0.0.1:8000 Hello, World! $ curl -i http://127.0.0.1:8000 HTTP/1.1 200 OK Server: nginx/1.18.0 (Ubuntu) Date: Sat, 03 Feb 2024 11:41:16 GMT Content-Type: application/octet-stream Content-Length: 14 Connection: keep-alive Hello, World! ``` As we can see the NGINX served the HTTP content when the request was made to port 8000 on the localhost. ## Serving from a custom path/folder But things are not like these in the real world, we need to serve an entire directory of HTML pages. We need to add the `root` directive with the path to the folder where our HTML content resides. The path should have the `index.html` file as the starting point of the request. ```nginx http { server { listen 8000; root /srv/techstructive-blog; index index.html; } } ``` **NOTE: The path to the HTML content needs to be accessible and the Nginx process should have the read permission to serve the contents.** It is commonly recommended to store HTML/static content files in directories such as `/srv` or `/var/www`. These paths follow conventions for serving static files and web applications in Unix-type operating systems. While it's not a strict requirement, adhering to these conventions can improve the organization and maintainability of web content. ## Serving from a web server If you already have a web server running in a port on your system, you could use Nginx as a gateway to the application instead of exposing your application to the internet. We could use the [proxy\_pass](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) directive in the location setting to specify which URL to pass the request to, the `listen` will forward the request to the proxy specified in the location directive. ```nginx http { server { listen 80; location / { proxy_pass http://localhost:8001; } } } ``` In the above example, the NGINX listens to port 80 in the local system and sends the request to the localhost at port 8001. The `proxy_pass` is used to specify the URL to redirect the request to. * **listen 80:** Nginx listens for incoming requests on port 80, the standard HTTP port. * **location /:** This directive matches all incoming requests, regardless of the path. * **proxy\_pass**[**http://localhost:8001**](http://localhost:8001)**:** Requests are forwarded to the web application running on [localhost](http://localhost) at port 8001. This example configuration is a basic building block for setting up more complex proxy configurations with NGINX. ## Serving from Multiple Upstream Servers NGINX can also serve content from multiple upstream servers, balancing the load between them. This is useful for high-traffic applications that require multiple backend servers to handle the load. What are upstream servers, you might ask, well in the context of NGINX, upstream servers refer to backend servers that handle the actual processing of requests. NGINX acts as a gateway, forwarding incoming requests to these upstream servers. This setup allows NGINX to manage the traffic efficiently and distribute it among multiple servers, which can be particularly beneficial for high-traffic applications. . For example, you might have your application running on [`localhost:8001`](http://localhost:8001) and [`localhost:8002`](http://localhost:8002). Here’s an example configuration: ```nginx http { upstream myapp { server backend1.example.com; server backend2.example.com; server backend3.example.com; } server { listen 80; location / { proxy_pass http://myapp; } } } ``` In this configuration: * The `upstream` block defines a named group of backend servers (`myapp`). * The `server` block listens on port 80 and proxies requests to the upstream group defined earlier. * `upstream myapp`: This directive creates a group of backend servers named `myapp`. * [`server backend1.example.com`](http://backend1.example.com) : These directives list the backend servers that will handle the requests. These can be specified by hostname, IP address, or combination. * `proxy_pass` [`http://myapp`](http://myapp): This directive tells NGINX to forward incoming requests to the `myapp` upstream group. ### Why Use Upstream Servers? Using upstream servers has several advantages: * Scalability: By distributing requests across multiple servers, you can handle more traffic and scale your application horizontally. * Fault Tolerance: If one of the backend servers goes down, NGINX can continue to serve requests using the remaining servers, ensuring high availability. * Load Distribution: Upstream servers help in balancing the load, which can improve the performance and responsiveness of your web application. The below configuration sets up NGINX to act as a gateway that distributes incoming traffic to multiple upstream servers. It defines an upstream block with servers at [`localhost:8001`](http://localhost:8001) and [`localhost:8002`](http://localhost:8002), and forward requests to these servers. ```nginx http { upstream myapp { server localhost:8001; server localhost:8002; } server { listen 80; location / { proxy_pass http://myapp; } } } ``` The provided NGINX configuration sets up an upstream block named `myapp` with two backend servers running on [`localhost`](http://localhost) at ports 8001 and 8002. The server block listens on port 80 and uses a location block to match all incoming requests to the root URL (`/`). These requests are forwarded to the `myapp` upstream group via the `proxy_pass` directive, allowing NGINX to distribute the requests between the two backend servers, effectively balancing the load and enhancing the application's performance and reliability. ## Conclusion From this part of ther series, we have learned how to set up a simple HTTP server to serve content from custom directories and using NGINX as a gateway to backend servers, which covered essential ways to utilize NGINX for serving web applications. That's it from this part of the series, we will look into detail how to use NGINX as a load balancer and reverse proxy, serving static files, and caching content in the next part of the series, where we'll dive deeper into advanced NGINX configurations. Thank you for reading, hopefully you found this helpful. If you have any feedback, questions, or queries drop them below in the comments or reach me out directly on my social handles. Happy Coding :) https://www.meetgor.com/posts/golang-web-put-method Introduction In this section of the series, we will be exploring how to send a HTTP request in golang. We will understand how to send a basic PUT request, creat Sat, 15 Jun 2024 00:00:00 UTC ## Introduction In this section of the series, we will be exploring how to send a `PUT` HTTP request in golang. We will understand how to send a basic PUT request, create an HTTP request, update a resource on a server, parsing the content from struct to json, headers, etc in the following section of this post. ## What is a PUT Method A PUT method is a type of request that is used to update or modify an entire resource on a server/database. Imagine you have ordered a pizza at a restaurant and realized you want to change the toppings after you've already placed the order. With a PUT request, it's like informing the waiter about the changes you want to make to your existing order. You specify the updated toppings or any modifications (the data you send). The waiter then takes this updated information (PUT request) back to the kitchen (the server) to apply the changes to your order. Let's say you created a order. ```nginx PUT /api/order/456 HTTP/1.1 Host: example.com Content-Type: application/json Content-Length: 123 { "userID": 123, "orderID": 456, "items": [ { "itemID": 789, "name": "Pizza", "quantity": 2, "toppings": ["Mushrooms"] } ] } ``` In the context of web development, PUT requests are often used for actions such as: * Updating existing records or resources * Modifying specific parts of an existing resource * Replacing an entire resource with updated data Here's an example of what the PUT request might look like in this scenario: ```nginx PUT /api/order/456 HTTP/1.1 Host: example.com Content-Type: application/json Content-Length: 155 { "userID": 123, "orderID": 456, "items": [ { "itemID": 789, "name": "Pizza", "quantity": 2, "toppings": ["Mushrooms", "Olives"] } ] } ``` In this example: * The PUT method is used to update the resource identified by `/api/order/456`. * The application/json is the content type of the request. * The 155 is the content length of the request. * The body contains the updated details of the order, including the addition of toppings to the pizza. PUT requests are crucial for maintaining and updating data in applications where accuracy and consistency are paramount, ensuring that resources are kept current and reflect the latest changes made by users or systems ## Why the need of PUT Method In the world of HTTP requests, we use the PUT method to update or modify an entire resource on a server or database. This method is crucial because the POST method, while convenient for creating new data, is not intended for updating existing resources according to standard conventions. While it's possible to misuse the POST method for updates internally, doing so can lead to confusion and inconsistencies in how requests are understood and processed. ## How PUT Method request works A [PUT](https://www.rfc-editor.org/rfc/rfc9110#PUT) request is utilized to send data to a server for the purpose of updating a resource. When a client (such as a browser or other APIs) sends a PUT request to the server's API endpoint, it includes data in the request body, typically formatted as JSON, XML, or form data. The server processes the PUT request by first identifying the resource to be updated, either through the URL or data provided in the request body. It then validates, parses, and applies the data from the request body to make modifications to the resource. Following this, the server returns a response that includes a status code indicating the success or failure of the operation. Optionally, the response may also include the updated resource in the response body. Unlike the POST method, which is primarily used for creating new resources, PUT is specifically designed for updating existing resources on the server. The request body of a PUT contains the data necessary for the update, while the URL identifies the specific resource to be updated. In summary, PUT requests facilitate the transfer of data to the server specifically for updating resources, ensuring that changes to existing data are accurately processed and reflected. ## Basic PUT Method To send a `PUT` request to an API in golang, we need to create a `http.Request` object. For `POST` method, the `http` package had the `Post` function defined, however for `PUT` method, there is no separate function. The Go philosophy is right now against adding all the method functions. There have been a couple of discussions on this on [GitHub](https://github.com/golang/go/issues/22841), but it is not been adopted as of 2024. So, we need to create a `http.Request` object for `PUT` method. ```go package main import ( "fmt" "net/http" ) func main() { apiURL := "https://reqres.in/api/users/5" req, err := http.NewRequest(http.MethodPut, apiURL, nil) if err != nil { panic(err) } resp, err := http.DefaultClient.Do(req) if err != nil { panic(err) } fmt.Println(resp.StatusCode) defer resp.Body.Close() } ``` ```bash $ go run main.go 200 ``` The above code sends a `PUT` request to the [`https://reqres.in/api/users/5`](https://reqres.in/api/users/5) endpoint. The resource we are trying to update is fetched with the identifier `5` which could probably be `id` for the user in the database of the server. ## PUT Method with JSON Marshaling and encoding are essential in Go for preparing structured data, such as from a struct, into JSON format suitable for HTTP requests like PUT. This conversion ensures data integrity and compatibility between Go types and JSON representations. It's crucial when updating resources on servers, as APIs often require specific data formats for processing updates correctly. Marshaling transforms Go structs into JSON bytes, while encoding further prepares them as request bodies, facilitating seamless communication with web services. This process ensures data consistency and adherence to API specifications, maintaining robust communication in distributed systems. ```go package main import ( "bytes" "encoding/json" "fmt" "io" "net/http" ) type User struct { Name string `json:"name"` Salary int `json:"salary"` Age string `json:"age"` ID int `json:"id,omitempty"` } type UserResponse struct { Status string `json:"status"` Message string `json:"message"` } func main() { user := User{ Name: "Alice", Salary: 50000, Age: "25", } apiURL := "https://dummy.restapiexample.com/api/v1/update/11" // marshalling process // converting Go specific data structure/types to JSON bodyBytes, err := json.Marshal(user) if err != nil { panic(err) } fmt.Println(string(bodyBytes)) // reading json into a buffer/in-memory body := bytes.NewBuffer(bodyBytes) // post request req, err := http.NewRequest(http.MethodPut, apiURL, body) if err != nil { panic(err) } resp, err := http.DefaultClient.Do(req) if err != nil { panic(err) } fmt.Println(resp.StatusCode) if resp.StatusCode == 429 { fmt.Println("too many requests") return } respBody, err := io.ReadAll(resp.Body) if err != nil { panic(err) } fmt.Println(string(respBody)) defer resp.Body.Close() // unmarshalling process // converting JSON to Go specific data structure/types var userResponse UserResponse if err := json.Unmarshal(respBody, &userResponse); err != nil { panic(err) } fmt.Println(userResponse) } ``` ```bash $ go run json.go {"name":"Alice","salary":50000,"age":"25"} 200 {"status":"success","data":[],"message":"Successfully! Record has been updated."} {success Successfully! Record has been updated.} ``` In the provided Go code example, the `json.Marshal` function is used to convert a Go struct (`User`) into a JSON formatted byte slice (`[]byte`). Here's a breakdown of the steps involved: * Struct Definition: Define a Go struct with json tags. * Marshalling: Use json.Marshal to convert the struct into JSON byte slice. * Buffer Creation: Wrap the JSON byte slice into an in-memory buffer (bytes.Buffer). * Request Sending: Send a PUT request with the buffer as the request body and set appropriate headers. Let's explore it step by step in detail: When using the PUT method in Go to update a resource on a server, you often need to send data in JSON format as the request body. Here's how you can achieve this using marshaling and encoding: 1. Define the Struct ```go type User struct { Name string `json:"name"` Salary int `json:"salary"` Age string `json:"age"` ID int `json:"id,omitempty"` } ``` Define a Go struct (`User`) that represents the data structure you want to send in JSON format. The json tags specify how each field should be serialized into JSON. 2. Create an Object ```go user := User{ Name: "Alice", Salary: 50000, Age: "25", } ``` Create an instance of the User struct (user) with sample data. This data will be marshaled into JSON format to send in the `PUT` request body. 3. Marshal the Struct ```go bodyBytes, err := json.Marshal(user) if err != nil { panic(err) } ``` Use json.Marshal(user) to convert the user struct into a JSON byte slice (bodyBytes). This byte slice contains the serialized JSON representation of the User struct. 4. Create a Buffer ```go body := bytes.NewBuffer(bodyBytes) ``` Use `bytes.NewBuffer(bodyBytes)` to create an in-memory buffer (`body`) containing the JSON byte slice (`bodyBytes`). The buffer implements the `io.Reader` interface needed for the PUT request body. 5. Create a PUT Request ```go req, err := http.NewRequest(http.MethodPut, apiURL, body) if err != nil { panic(err) } ``` Use http.NewRequest to create a new PUT request to the specified URL with the JSON buffer (`body`) as the request body. Set appropriate headers if needed (e.g., Content-Type as application/json). 6. Send the Request ```go resp, err := http.DefaultClient.Do(req) if err != nil { panic(err) } ``` Use [`http.DefaultClient.Do`](http://http.DefaultClient.Do)`(req)` to execute the PUT request and obtain the response. Handle any errors that may occur during the request execution. 7. Process the Response ```go respBody, err := io.ReadAll(resp.Body) if err != nil { panic(err) } ``` Use `io.ReadAll(resp.Body)` to read and store the response body from the server. Handle any errors encountered during the reading process. 8. Unmarshal the Response ```go var userResponse UserResponse if err := json.Unmarshal(respBody, &userResponse); err != nil { panic(err) } ``` Use `json.Unmarshal(respBody, &userResponse)` to deserialize the JSON response body into a Go struct. This allows you to work with the response data in a structured manner. The parsing of files and form data is also possible with `PUT` requests, however, that has been covered in the [POST Method](https://meetgor.com/golang-web-post-method). Those snippets would be handy in these request method as well. I have also included some more examples of PUT requests [here](https://github.com/Mr-Destructive/100-days-of-golang/blob/main/web/methods/put/). That's it from the 35th part of the series, all the source code for the examples are linked in the GitHub on the [100 days of Golang](https://github.com/Mr-Destructive/100-days-of-golang/tree/main/web/methods/put/) repository. [100-days-of-golang](https://github.com/Mr-Destructive/100-days-of-golang) ## Conclusion That's it from this post of the series, a post on the PUT method in golang :) We have covered topics like creating basic PUT requests and marshaling golang types into JSON format. Hope you found this article helpful. If you have any queries, questions, or feedback, please let me know in the comments or on my social handles. Thank you for reading. Happy Coding :) https://www.meetgor.com/posts/golang-web-post-method Introduction In this section of the series, we will be exploring how to send a HTTP request in golang. We will understand how to send a basic POST request, crea Sun, 10 Mar 2024 00:00:00 UTC ## Introduction In this section of the series, we will be exploring how to send a `POST` HTTP request in golang. We will understand how to send a basic POST request, create an HTTP request, and parse json, structs into the request body, add headers, etc in the following sections of this post. We will understand how to marshal the golang struct/types into JSON format, send files in the request, and handle form data with examples of each in this article. Let's answer a few questions first. ## What is a POST request? POST method is a type of request that is used to send data to a server(a machine on the internet). Imagine you are placing an order at a restaurant. With a GET request, it would be like asking the waiter, "What kind of pizza do you have?" The waiter would respond by telling you the menu options (the information retrieved from the server). However, a POST request is more like giving your completed order to the waiter. You tell them the specific pizza you want, its size, and any additional toppings (the data you send). The waiter then takes this information (POST request) back to the kitchen (the server) to process it (fulfill your order). In the world of web development, POST requests are often used for things like: Submitting forms (e.g., contact forms, login forms) Uploading files (e.g., photos, videos) Creating new accounts Sending data to be processed (e.g., online purchases) Here's an example of what the POST request might look like in this scenario: ```nginx POST /api/order HTTP/1.1 Host: example.com Content-Type: application/json Content-Length: 123 { "userID": 123, "orderID": 456, "items": [ { "itemID": 789, "name": "Pizza", "quantity": 2 }, { "itemID": 999, "name": "Burger", "quantity": 1 } ] } ``` In this example: * The `POST` method is used to send data to the server. * The `/api/order` is the endpoint of the server. * The `application/json` is the content type of the request. * The `123` is the content length of the request. * The `{"userID": 123, "orderID": 456, "items": [{"itemID": 789, "name": "Pizza", "quantity": 2}, {"itemID": 999, "name": "Burger", "quantity": 1}]}` is the body of the request. ## Why the need for a POST request? In the world of HTTP requests, we use the POST method to securely send data from a client (like a user's browser) to a server. This is crucial because the GET method, while convenient for retrieving data, has limitations: Imagine you are in registering for an event via Google form, you type in your details on the webpage like name, email, address, phone number, and other personal details. If the website/app was using the `GET` method to send the request to register or do any other authentication/privacy-related requests, it could expose the data in the URL itself. It would be something along the lines [`https://form.google.com/register/<form-name>-<id>/?name=John&phone_number=1234567890`](https://form.google.com/register/%3Cform-name%3E-%3Cid%3E/?name=John&phone_number=1234567890), if a user maliciously sniffs into your network and inspects the URL, your data will be exposed. That is the reason we need `POST` a method. ## How a POST method works? A [POST](https://www.rfc-editor.org/rfc/rfc9110#POST) request is used to send data to a server to create or update(there is a separate method for updating) a resource. The client(browser/other APIs) sends a POST request to the server's API endpoint with the data in the request body. This data can be in formats like JSON, XML, or form data. The server processes the POST request, validates and parses the data in the request body, makes any changes or creates resources based on that data, and returns a response. The response would contain a status code indicating the success or failure of the operation and may contain the newly created or updated resource in the response body. The client must check the response status code to verify the outcome and process the response accordingly. Unlike GET, POST can create new resources on the server. The body of a POST contains the data for creation while the URL identifies the resource to be created. Overall, POST transfers data to the server for processing, creation or updating of resources. The status code is usually `201` indicating the resource is successfully created or `200` for just indicating success. Some common steps for creating and sending a POST request as a developer include: * Defining the API endpoint * Clarifying the data format (json, language native objects, xml , text, form-data, etc) * Converting / Marshalling the data * Attaching header for `Content-Type` as key and value as the format of the data type (e.g. `application/json` for json) * Sending the request The above steps are general for creating and sending a POST request, they are not specific to Golang. For golang specific steps, we need to dive a bit deeper, let's get started. ## Basic POST method in Golang To send a POST request in golang, we need to use the `http` package. The `http` package has the `Post` method, which takes in 3 parameters, namely the URL, the Content-Type, and the Body. The body can be `nil` if the URL endpoint doesn't necessarily require a body. The `Content-Type` is the string, since we are just touching on how the Post request is constructed, we will see what the `Content-Type` string value should be in the later sections. > [http.Post](http://http.Post)(URL, Content-Type, Body) ```go package main import ( "fmt" "net/http" ) func main() { apiURL := "https://reqres.in/api/users" // POST request resp, err := http.Post(apiURL, "", nil) // ideally the Content-Type header should be set to the relevant format // resp, err := http.Post(apiURL, "application/json", nil) if err != nil { panic(err) } fmt.Println(resp.StatusCode) fmt.Println(resp) defer resp.Body.Close() } ``` ```bash $ go run main.go 201 &{ 201 Created 201 HTTP/2.0 2 0 map[ Access-Control-Allow-Origin:[*] Cf-Cache-Status:[DYNAMIC] Cf-Ray:[861cd9aec8223e4b-BOM] Content-Length:[50] Content-Type:[application/json; charset=utf-8] Date:[Sat, 09 Mar 2024 17:40:28 GMT] Server:[cloudflare] ... ... ... X-Powered-By:[Express] ] {0xc00017c180} 50 [] false false map[] 0xc000156000 0xc00012a420 } ``` The above code is sending the `POST` request to the [`https://reqres.in/api/users`](https://reqres.in/api/users) endpoint with an empty body and no specific format for `Content-Type` header. The response is according to the [Response](https://pkg.go.dev/net/http#Response) structure. We can see we got `201` status, which indicates the server received the POST request successfully, the API is a dummy api, so we don't care about the data we are processing, we are just using the API as a placeholder for sending the POST request. We can use `map[string]interface{}` it to pass the data in the request body. The `json.Marshal` method is used to convert the map into JSON format. We will look into the details shortly in the next few examples. ```go package main import ( "bytes" "encoding/json" "fmt" "net/http" ) func main() { apiURL := "https://reqres.in/api/users" bodyMap := map[string]interface{}{ "name": "morpheus", "job": "leader", } requestBody, err := json.Marshal(bodyMap) if err != nil { panic(err) } body := bytes.NewBuffer(requestBody) resp, err := http.Post(apiURL, "application/json", body) if err != nil { panic(err) } fmt.Println(resp.StatusCode) defer resp.Body.Close() } ``` ```bash $ go run main.go 201 ``` The above code sends the `POST` request to the [`https://reqres.in/api/users`](https://reqres.in/api/users) endpoint with the data in the request body in JSON format. ## Creating a POST request in Golang We can construct the POST request with the [NewRequest](https://pkg.go.dev/net/http#NewRequest) method. The method takes in 3 parameters, namely the `method` (e.g. `POST`, `GET`), the `URL` and the `body` (if there is any). We can then add extra information to the headers or the Request object after constructing the basic HTTP [Request](https://pkg.go.dev/net/http#Request) object. ```go package main import ( "fmt" "net/http" ) func main() { apiURL := "https://reqres.in/api/users" req, err := http.NewRequest(http.MethodPost, apiURL, nil) if err != nil { panic(err) } resp, err := http.DefaultClient.Do(req) if err != nil { panic(err) } fmt.Println(resp.StatusCode) //fmt.Println(resp) defer resp.Body.Close() } ``` ```bash $ go run main.go 201 ``` In the above example, we have created an HTTP Request as the `POST` method, with [`https://reqres.in/api/users`](https://reqres.in/api/users) as the URL, and no body. This constructs an HTTP Request object, which can be sent as the parameter to the [http.DefaultClient.Do](http://http.DefaultClient.Do) method, which is the default client for the request we sent in the earlier examples as `http.Get` or [`http.Post`](http://http.Post) methods. We can implement a custom client as well, and then apply `Do` the method with the request parameters. The `Do` method returns the `Request` object or the `error` if any. More on the customizing Client will be explained in a separate post in the series. The response is also in the same format as the [Response](https://pkg.go.dev/net/http#Response) structure that we have seen earlier. This section of the series aims to construct a post request, and not to parse the response, we have already understood the parsing of the response in the [Get method](https://www.meetgor.com/golang-web-get-method/#?:~:text=Parsing%20the%20JSON%20body%20with%20structs) section of the series. ### Parsing objects to JSON for POST method request We might have a golang object that we want to send as a body to an API in the POST request, for that we need to convert the golang struct object to JSON. We can do this by using the [Marshal](https://pkg.go.dev/encoding/json#Marshal) or the [Encode](https://pkg.go.dev/encoding/json#Encoder.Encode) method for serialization of the golang struct object to JSON. #### Using Marshal method Marshaling is the process of converting data from a data structure into a format suitable for transmission over a network or for storage. It's commonly used to convert native objects in a programming language into a serialized format, typically a byte stream, that can be transmitted or stored efficiently. You might get a question here, what is the difference between `Marshalling` and `Serialization`? Well, Serialization, is a broader term that encompasses marshalling. It refers to the process of converting an object or data structure into a format that can be stored or transmitted and later reconstructed into the original object. Serialization may involve converting data into byte streams, XML, JSON, or other formats. So, in summary, marshaling specifically deals with converting native objects into a format suitable for transmission, while serialization encompasses the broader process of preparing data for storage or transmission. The `json` package has the [Marshal](https://pkg.go.dev/encoding/json#Marshal) method that converts the golang object into JSON. The `Marshal` method takes in a parameter as the struct object with type `any` and returns a byte slice `[]byte` and error (if any). ```go package main import ( "bytes" "encoding/json" "fmt" "net/http" ) type User struct { Name string `json:"name"` Salary int `json:"salary"` Age int `json:"age"` } func main() { user := User{ Name: "Alice", Salary: 50000, Age: 25, } apiURL := "https://dummy.restapiexample.com/api/v1/create" // marshalling process // converting Go specific data structure/types to JSON bodyBytes, err := json.Marshal(user) if err != nil { panic(err) } fmt.Println(string(bodyBytes)) // reading json into a buffer/in-memory body := bytes.NewBuffer(bodyBytes) // post request resp, err := http.Post(apiURL, "application/json", body) if err != nil { panic(err) } fmt.Println(resp.StatusCode) defer resp.Body.Close() } ``` ```bash $ go run main.go {"name":"Alice","salary":50000,"age":25} 200 ``` In the above example, we have created a struct `User` with fields `Name`, `Salary`, and `Age`, the json tags will help label each key in JSON with the tag for the respective fields in the struct. We create an object `user` of a type `User` with the values as `Alice`, `50000`, and `25` respectively. We call the `json.Marshal` method with the parameter `user` that represents the struct object `User`, the method returns a slice of bytes, or an error either or both could be nil. If we try to see the stringified representation of the byte slice, we can see something like `{"name":"Alice","salary":50000,"age":25}` which is a JSON string for the user struct. We can't parse the byte slice as the body in the POST request, we need the `io.Reader` object, so we can load the byte slice `bodyBytes` into a buffer and parse that as a body for the POST request. We then send a `POST` request to the endpoint [`https://dummy.restapiexample.com/api/v1/create`](https://dummy.restapiexample.com/api/v1/create) with the content type as `application/json` and with the body as `body` which was a `io.Reader` object as an in-memory buffer. In brief, we can summarize the marshaling of the golang object into JSON with `Marshal` function as the following steps: * Defining the structure as per the request body * Creating the struct object for parsing the data as body to the request * Calling the `json.Marshal` function to convert the object to JSON (parameter as the struct object `any` type) * Loading the byte slice into a buffer with `bytes.NewBuffer()` * Sending the POST request to the endpoint with the body as the `io.Reader` object and content type as `application/json` #### Using Encode method We can even use the [Encoder.Encode](https://pkg.go.dev/encoding/json#Encoder.Encode) method to parse the golang struct object to JSON. Firstly, we should have the struct defined as per the request body that the particular API takes, we can make use of the json tags, omitempty, omit(-) options to make the marshaling process work accordingly. We can then create the object of that particular struct with the data we require to be created as a resource with the POST request on that API service. Thereafter we can create an empty buffer object with [bytes.Buffer](https://pkg.go.dev/bytes#Buffer), this buffer object would be used to initialize the [Encoder](https://pkg.go.dev/encoding/json#Encoder) object with the [NewEncoder](https://pkg.go.dev/encoding/json#NewEncoder) method. This would give access to the [Encode](https://pkg.go.dev/encoding/json#Encoder.Encode) method, which is used to take in the struct object (`any` type) and this will populate the buffer we initialized with the `NewEncoder` method. Later we can access that buffer to parse it to the Post request as the body. Let's understand it better with an example. ```go package main import ( "bytes" "encoding/json" "fmt" "net/http" ) type User struct { Name string Salary int Age int } func main() { user := User{ Name: "Alice", Salary: 50000, Age: 25, } apiURL := "https://dummy.restapiexample.com/api/v1/create" var bodyBuffer bytes.Buffer var encoder = json.NewEncoder(&bodyBuffer) err := encoder.Encode(user) if err != nil { panic(err) } resp, err := http.Post(apiURL, "application/json", &bodyBuffer) if err != nil { panic(err) } fmt.Println(resp.StatusCode) fmt.Println(resp) defer resp.Body.Close() } ``` Over here, we have created a struct `User` with fields `Name`, `Salary`, and `Age`, we initialize the `user` as the object of the `User` struct. Then we create a buffer `bodyBuffer` of type `bytes.Buffer` this is the actual buffer that we will send as the body. Further, we initialize the `Encoder` object as `encoder` with the `json.NewEncoder` method by parsing the reference of `bodyBuffer` as the parameter. Since `bytes.Buffer` implements the `io.Writer` interface, we can pass the `bodyBuffer` to the `NewEncoder` method. This will create the `Encoder` object which in turn will give us access to the `Encode` method, where we will parse the struct instance and it will populate the buffer with which we initialized the `Encoder` object earlier. Now, we have the `encode` object, this gives us the access to `Encode` method, we call the `Encode` method with the parameter of `user` which is a User struct instance/object. The Encode method will populate the `bodyBuffer` object or it will result in an error if anything goes wrong (the data is incorrectly parsed or is not in the required format). We can call the `Post` method with the initialized URL, the `Content-Type` as `application/json` since we have converted the struct instance to JSON object, and the body as the reference to the buffer as `&bodyBuffer` So, the steps for parsing struct instances into JSON objects with the `Encoder.Encode` method is as follows: * Defining the structure as per the request body * Creating the struct object for parsing the data as body to the request * Creating an empty `bytes.Buffer` object as an in-memory buffer * Initializing the `Encoder` object with `NewEncoder` method by parsing the reference of `bodyBuffer` as the parameter * Calling the `Encode` method with the parameter of struct instance/object * Sending the POST request to the endpoint with the content type as `application/json` and body as the reference to the buffer The results are the same as the above example just the way we have parsed the struct instance to JSON object is different. ### Parsing JSON to POST request We have seen how we can parse golang struct instances to JSON and then send the post request, but what if we had the JSON string already with us, and we want to send the request? Well, that's much easier, right? We already have parsed the JSON string to the Post request by loading the slice of bytes into a buffer, so we just need to convert the string to a slice of bytes which is quite an easy task, and then load that byte slice to the buffer. ```go package main import ( "bytes" "fmt" "net/http" ) func main() { // dummy api apiURL := "https://dummy.restapiexample.com/api/v1/create" // json data data := `{ "name": "Alice", "job": "Teacher" }` body := bytes.NewBuffer([]byte(data)) // POST request resp, err := http.Post(apiURL, "application/json", body) if err != nil { panic(err) } fmt.Println(resp.StatusCode) fmt.Println(resp) defer resp.Body.Close() } ``` In the example above, we already have a JSON string `data` with keys as `name` and `job` but it is not JSON, it is a stringified JSON. We can convert the stringified JSON to a slice of bytes using the `[]byte` function. Further, we have used the `bytes.NewBuffer` method to load the byte slice into an `io.Reader` object. This object returned by the `bytes.NewBuffer` will serve as the body for the POST request. ### Parsing JSON to objects in Golang from POST method response ```go package main import ( "bytes" "encoding/json" "fmt" "io" "net/http" ) type User struct { Name string `json:"name"` Salary int `json:"salary"` Age string `json:"age"` ID int `json:"id,omitempty"` } type UserResponse struct { Status string `json:"status"` Data User `json:"data"` } func main() { user := User{ Name: "Alice", Salary: 50000, Age: "25", } apiURL := "https://dummy.restapiexample.com/api/v1/create" // marshalling process // converting Go specific data structure/types to JSON bodyBytes, err := json.Marshal(user) if err != nil { panic(err) } fmt.Println(string(bodyBytes)) // reading json into a buffer/in-memory body := bytes.NewBuffer(bodyBytes) // post request resp, err := http.Post(apiURL, "application/json", body) if err != nil { panic(err) } fmt.Println(resp.StatusCode) fmt.Println(resp) defer resp.Body.Close() // Read response body respBody, err := io.ReadAll(resp.Body) if err != nil { panic(err) } // unmarshalling process // converting JSON to Go specific data structure/types var userResponse UserResponse if err := json.Unmarshal(respBody, &userResponse); err != nil { panic(err) } fmt.Println(userResponse) fmt.Println(userResponse.Data) } ``` ```nginx {success {Alice 50000 25 3239}} {Alice 50000 25 577} ``` The above example is a POST request with a struct instance being loaded as a JSON string and then sent as a buffer to the API endpoint, it also reads the response body with a specific structure `UserResponse` and unmarshalled the `resp.Body` from the `io.Reader` as `respBody` and then loads into `userResponse` object. This example gives an entire process of what we have understood in the JSON data parsing for a POST request. ### Sending Form data in a POST request We can also send data to a POST request in the form of a form, the form which we use in the HTML. Golang has a `net/url` package to parse the form data. The form data is sent in the `application/x-www-form-urlencoded` format. ```go package main import ( "encoding/json" "fmt" "io" "net/http" "net/url" "strings" ) type ResponseLogin struct { Token string `json:"token"` } func main() { // dummy api apiURL := "https://reqres.in/api/login" // Define form data formData := url.Values{} formData.Set("email", "eve.holt@reqres.in") formData.Set("password", "cityslicka") // Encode the form data fmt.Println(formData.Encode()) reqBody := strings.NewReader(formData.Encode()) fmt.Println(reqBody) // Make a POST request with form data resp, err := http.Post(apiURL, "application/x-www-form-urlencoded", reqBody) if err != nil { panic(err) } defer resp.Body.Close() // Print response status code fmt.Println("Status Code:", resp.StatusCode) // Read response body respBody, err := io.ReadAll(resp.Body) if err != nil { panic(err) } token := ResponseLogin{} json.Unmarshal(respBody, &token) fmt.Println(token) } ``` ```bash $ go run main.go email=eve.holt%40reqres.in&password=cityslicka &{email=eve.holt%40reqres.in&password=cityslicka 0 -1} Status Code: 200 {QpwL5tke4Pnpja7X4} ``` In the above example, we set a `formData` with the values of `email` and `password` which are `url.Values` object. The `url.Values` the object is used to store the key-value pairs of the form data. The `formData` is encoded with the `url.Encode` method, We load the encoded string to a `buffer` with `strings.NewReader` which implements the `io.Reader` interface, so that way we can pass that object as the body to the post request. We send the `POST` request to the endpoint [`https://reqres.in/api/login`](https://reqres.in/api/login) with the content type as `application/x-www-form-urlencoded` and with the body as `reqBody` which implements the `io.Reader` interface as an in-memory buffer. The response from the request is read into the buffer with `io.ReadAll` method and we can `Unmarshal` the stream of bytes as a buffer into the `ResponseLogin` struct object. The output shows the `formData` as encoded string `email=eve.holt%`[`40reqres.in`](http://40reqres.in)`&password=cityslicka` as `@` is encoded to `%40`, then we wrap the `formData` in a `strings.NewReader` object which is a buffer that implements `io.Reader` interface, hence we can see the result as the object. The status code for the request is `200` indicating the server received the `form-data` in the body and upon unmarshalling, we get the token as a response to the POST request which was a dummy login API. This way we have parsed the form-data to the body of a POST request. ### Sending File in a POST request We have covered, parsing text, JSON, and form data, and now we need to move into sending files in a POST request. We can use the `multipart` package to parse files into the request body and set appropriate headers for reading the file from the API services. We first read the file contents [`os.Open`](http://os.Open) which returns a reference to the `file` object or an error. We create an empty `bytes.Buffer` object as `body` which will be populated later. The [multipart.NewWriter](https://pkg.go.dev/mime/multipart#NewWriter) method takes in the `io.Writer` object which will be the `body` as it is an `bytes.Buffer` object that implements the `io.Writer` interface. This will initialize the [Writer](https://pkg.go.dev/mime/multipart#Writer) object in the `multipart` package. We create a `form-field` in the `Writer` object with the [CreateFormFile](https://pkg.go.dev/mime/multipart#Writer.CreateFormFile) method, which takes in the `fieldName` as the name of the field, and the `fileName` as the name of the file which will be read later in the multipart form. The method returns either the part or the error. The `part` is an object that implements the `io.Writer` interface. Since we have stored the file contents in the `file` object, we copy the contents into the `form-field` with the [Copy](https://pkg.go.dev/io#Copy) method. Since the `part` return from the `CreateFormFile` was implementing the `io.Writer` interface, we can use it to Copy the contents from source to destination. The source is the `io.Reader` object and the destination is the `io.Writer` object, the destination for the `Copy` method is the first parameter, the source is the second parameter. This Copy method will populate the buffer initialized earlier in the `NewWriter` method. This will give us a buffer that has the file contents in it. We can pass this buffer to the POST request with the `body` parameter. We also need to make sure we close the `Writer` object after copying the contents of the file. We can extract the type of file which will serve as the `Content-Type` of the request. Let's clear the explanation with an example. ```go package main import ( "bytes" "encoding/json" "fmt" "io" "mime/multipart" "net/http" "os" ) type ResponseFile struct { Files map[string]string `json:"files"` } func main() { apiURL := "http://postman-echo.com/post" fileName := "sample.csv" file, err := os.Open(fileName) if err != nil { panic(err) } defer file.Close() body := &bytes.Buffer{} writer := multipart.NewWriter(body) part, err := writer.CreateFormFile("csvFile", fileName) if err != nil { panic(err) } _, err = io.Copy(part, file) if err != nil { panic(err) } contentType := writer.FormDataContentType() fmt.Println(contentType) writer.Close() resp, err := http.Post(apiURL, contentType, body) if err != nil { panic(err) } defer resp.Body.Close() fmt.Println("Status Code:", resp.StatusCode) respBody, err := io.ReadAll(resp.Body) if err != nil { panic(err) } token := ResponseFile{} json.Unmarshal(respBody, &token) fmt.Println(token) fmt.Println(token.Files[fileName]) } ``` ```bash multipart/form-data; boundary=7e0eacfff890be395eba19c70415c908124b503a56f23ebeec0ab3c665ca --619671ea2c0aa47ca6664a7cda422169d73f3b8a089c659203f5413d03de Content-Disposition: form-data; name="csvFile"; filename="sample.csv" Content-Type: application/octet-stream User,City,Age,Country Alex Smith,Los Angeles,20,USA John Doe,New York,30,USA Jane Smith,Paris,25,France Bob Johnson,London,40,UK --619671ea2c0aa47ca6664a7cda422169d73f3b8a089c659203f5413d03de-- Status Code: 200 {map[sample.csv:data:application/octet-stream;base64,VXNlcixDaXR5LEFnZSxDb3VudHJ5CkFsZXggU21pdGgsTG9zIEFuZ2VsZXMsMjAsVVNBCkpvaG4gRG9lLE5ldyBZb3JrLDMwLFVTQQpKYW5lIFNtaXRoLFBhmlzLDI1LEZyYW5jZQpCb2IgSm9obnNvbixMb25kb24sNDAsVUsK]} data:application/octet-stream;base64,VXNlcixDaXR5LEFnZSxDb3VudHJ5CkFsZXggU21pdGgsTG9zIEFuZ2VsZXMsMjAsVVNBCkpvaG4gRG9lLE5ldyBZb3JrLDMwLFVTQQpKYW5lIFNtaXRoLFBhmlzLDI1LEZyYW5jZQpCb2IgSm9obnNvbixMb25kb24sNDAsVUsK ``` In the above example, we first read the file `sample.csv` into the `file` object with [`os.Open`](http://os.Open) method, this will return a reference to the file object or return an error if any arises while opening the file. Then we create an empty buffer `bytes.Buffer` object which will serve as the body of the post request later as it will get populated with the file contents in the form of `multipart/form-data`. We initialize the `Writer` object with `multipart.NewWriter` method which takes in the empty buffer as the parameter, we parse the `body` as the parameter. The method will return a reference to the `multipart.Writer` object. With the `Writer` object we access the `CreateFormFile` method which takes in the `fieldName` as the name of the field, and the `fileName` as the name of the file. The method will return either the part or an error. The `part` in this case, is the reference to the `io.Writer` object that will be used to write the contents from the uploaded file. Then, we can use the `io.Copy` method to copy the contents from the `io.Reader` object to the `io.Writer` object. The source is the `io.Reader` object and the destination is the `io.Writer` object. The first parameter is however the destination and the second parameter is the source. In the example, we call `io.Copy(part, file)` which will copy the contents of `file` to the `part` buffer. We get the `Content-Type` by calling the [Writer.FormDataContentType](https://pkg.go.dev/mime/multipart#Writer.FormDataContentType) method. This returns us `multipart/form-data; boundary=7e0eacfff890be395eba19c70415c908124b503a56f23ebeec0ab3c665ca` which will serve the `Content-Type` for the Post request. We need to make sure we close the `Writer` object with the `Close` method. We just print the `body.String()` to get a look at what the actual body looks like, we can see there is a form for the file as a `form-data` with keys like `Content-Type`, `Content-Disposition`, etc. The file has the `Content-Type` as `application/octet-stream` and the actual content is rendered in the output. The dummy API responds with a 200 status code and also sends the JSON data with the name of the file as the key and the value as the `base64` encoded value of the file contents. This indicates that we were able to upload the file to the server API using a POST request. Well done! I have also included some more examples of POST requests with files [here](https://github.com/Mr-Destructive/100-days-of-golang/blob/main/web/methods/post/file_2.go) which extends the above example by taking the encoded values and decoding to get the actual contents of the file back. ## Best Practices for POST method Here are some of the best practices for the POST method which are followed to make sure you consume or create the POST request in the most secure, efficient, and graceful way. ### Always Close the Response Body Ensure that you close the response body after reading from it. Use `defer response.Body.Close()` to automatically close the body when the surrounding function returns. This is crucial for releasing associated resources like network connections or file descriptors. Failure to close the response body can lead to memory leaks, particularly with a large volume of requests. Properly closing the body prevents resource exhaustion and maintains efficient memory usage. ### Client Customization Utilize the [Client](https://pkg.go.dev/net/http#Client) struct to customize the HTTP client behavior. By using a custom client, you can set timeouts, headers, user agents, and other configurations without modifying the `DefaultClient` provided by the `http` package. This approach allows for flexibility and avoids repetitive adjustments to the client configuration for each request. ### Set Content-Type Appropriately Ensure that you set the `Content-Type` header according to the request payload. Correctly specifying the Content-Type is crucial for the server to interpret the request payload correctly. Failing to set the Content-Type header accurately may result in the server rejecting the request. Always verify and match the Content-Type header with the content being sent in the POST request to ensure smooth communication with the server. ## Reference * [Postman POST API](https://www.postman.com/postman/workspace/postman-answers/documentation/13455110-00378d5c-5b08-4813-98da-bc47a2e6021d) (For POST request with file upload) * [Golang net/http Package](https://pkg.go.dev/net/http) ## Conclusion That's it from this post of the series, a post on the POST method in golang :) We have covered topics like creating basic post requests, Marshalling golang types into JSON format, parsing form data, sending a POST request with files, and best practices for the POST method. Hope you found this article helpful. If you have any queries, questions, or feedback, please let me know in the comments or on my social handles. Happy Coding :) https://www.meetgor.com/posts/nginx-01-basics Introduction NGINX is a tool that can be used as a web server, reverse proxy, load balancer, streaming media files, application gateway, content caching, and so Sun, 14 Jan 2024 00:00:00 UTC ## Introduction NGINX is a tool that can be used as a web server, reverse proxy, load balancer, streaming media files, application gateway, content caching, and so much more. It can be said to be a Swiss army knife for optimizing and securing your web application deployment. The series "NGINX Survival Guide" will start from the basics and cover the bare minimum required for a backend developer to get going with NGINX. I will use Docker widely throughout this course as it is a great combination with NGINX to server web applications. However, you can use NGINX without docker, and spawn multiple servers. The series will cover the terminologies of NGINX, configuring NGINX servers, load balancing multiple servers, using it as a reverse proxy, and as an API gateway, there will be tiny details and some tidbits of doing certain things in a certain constrained environment which will make the learning more valuable. ## What is NGINX NGINX (pronounced "engine-x") is not just a web server, it is a powerful and versatile open-source software that wears many hats in the internet world. At its core, it functions as a **lightning-fast web server**, its secret weapon lies in its **event-driven architecture**, where it handles requests asynchronously, allowing it to serve countless users simultaneously without breaking a sweat. > NGINX is a popular choice for powering some of the **biggest websites and platforms in the world**, demonstrating its reliability and scalability. NGINX's **configurable nature** lets you tailor its behavior to your specific needs, whether managing traffic flow with load balancing, caching frequently accessed content for faster delivery, or even acting as a gateway for your APIs. This versatility makes NGINX a **powerful tool for building efficient, secure, and scalable web applications**, regardless of size or complexity. Hence the need to learn it as a developer and especially important for a backend developer. ### Why NGINX is must learn for backend developers Nginx is a highly efficient and performant web server. Understanding its configuration and management allows a backend developer to optimize server performance, handle large volumes of traffic, and reduce latency. In microservices architectures, Nginx can serve as an API gateway, managing and routing requests between different services. Nginx provides caching mechanisms that enhance performance by serving cached content, reducing the load on backend servers. Having strong fundamentals in NGINX can indeed provide a competitive edge in the job market and make a backend developer more versatile in handling various aspects of backend web development. ### Who is using NGINX? Big Tech Companies are using NGINX like DropBox, Netfilx, and Cloudflare, among others. Cloudflare used NGINX before but it was not enough for them, so they developed their web server/edge proxy suited to their needs called Pingora. * Dropbox - [Optimizing web servers for high throughput and low latency](https://dropbox.tech/infrastructure/optimizing-web-servers-for-high-throughput-and-low-latency) * Cloudflare - [How Cloudflare outgrown NGINX and made way to Pingora](https://blog.cloudflare.com/how-we-built-pingora-the-proxy-that-connects-cloudflare-to-the-internet/) * Netflix - [NGINX Netflix archives](https://www.nginx.com/blog/tag/netflix/) ## Installing NGINX ### Linux There are comprehensive guides for your specific flavor/package manager/preferences in the [official documentation](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-open-source/) of NGINX. A couple of common types of installation medium instructions are as follows: ```bash # APT sudo apt update sudo apt install nginx # YUM sudo yum install epel-release sudo yum update sudo yum install nginx ``` Check the status of the NGINX service to ensure the installation was successful or not with the command: ```bash sudo systemctl status nginx ``` If this doesn't have any errors or fatal messages, the nginx server is up and running on port 80 i.e. on `127.0.0.1` on the system. ### MacOS The installation on MacOS for NGINX is pretty simple with homebrew. The following [article](https://dev.to/arjavdave/installing-nginx-on-mac-46ac) walks through the steps of the installation: ```bash brew update brew install nginx nginx ``` If you do not want to install it from homebrew, this [gist](https://gist.github.com/beatfactor/a093e872824f770a2a0174345cacf171) can help install it from the source. ### Windows For Windows installation, you can follow the [guide](https://nginx.org/en/docs/windows.html) from the official documentation. ```bash # INSTALL the https://nginx.org/en/download.html # A Zip file with the name nginx-version.zip will be downlaoded # COPY it to the desired location and use that path while unzipping cd c:\ unzip nginx-1.25.3.zip cd nginx-1.25.3 start nginx ``` You can check the status of NGINX if the installation was successful or not with the command: ```bash tasklist /fi "imagename eq nginx.exe" ``` This should be from the installation section. ## Understanding the default config When you have completed the installation of nginx, you can see the default nginx configuration in the file path as `/etc/nginx/nginx.conf` in Linux/macOS or `C: ginx https://www.meetgor.com/posts/neovim-sourcegraph-cody Introduction Have you ever used Sourcegraph's Cody? It is a great tool for developers, it is not just another LLM, it is tailored specifically for developers. C Sat, 06 Jan 2024 00:00:00 UTC ## Introduction Have you ever used Sourcegraph's Cody? It is a great tool for developers, it is not just another LLM, it is tailored specifically for developers. Cody has some good features that allow parsing of context for the prompt in a smarter way. ### What is Sourcegraph's Cody Cody is an AI assistant for developers that understands code context and can generate code. It goes beyond just answering questions - it can write code for you. The major features of Cody from the rest of the LLMs or Chatbots are: * Cody understands your code context - it reads your open files, repositories, etc. So it can answer questions specifically about your codebase, not just general programming questions. * Cody can explain sections of code to you in plain English. This helps ramp up on unfamiliar code bases. * Cody integrates into popular editors like VS Code, IntelliJ, Neovim, and others for frictionless use while coding. For more insights, check out the blog [all you need is Cody](https://about.sourcegraph.com/blog/all-you-need-is-cody). This is a great article about what and how Cody is tailored specifically to assist developers. ## Prerequisites To setup sourcegraph on neovim, you will require the following setup: * Neovim 0.9 or above * Node.js &gt;= 18.17 (LTS) * Cargo (Rust) (optional) To install neovim latest/nightly release, you can follow the [INSTALL](https://github.com/neovim/neovim/blob/master/INSTALL.md) or [BUILD](https://github.com/neovim/neovim/blob/master/BUILD.md) documentation of the neovim project. Cargo is optional, as the plugin will install the binaries itself, however, if you prefer to have cargo, just install it in case something goes wrong. ## Installing sg.nvim There is a specific plugin for neovim for interacting with the Sourcegraph products, and Cody is one of them. The [sg.nvim](https://github.com/sourcegraph/sg.nvim) is a plugin for integrating sourcegraph search, Cody, and other features provided by soucegraph. #### Using packer.nvim ```lua use { 'sourcegraph/sg.nvim', run = 'nvim -l build/init.lua' } use { 'nvim-lua/plenary.nvim' } ``` Source your lua file where you have configured all the plugins and then enter the command `:PackerInstall` or `:PackerSync` to install the plugin. #### Using vim-plug If you are using vim-plug as the plugin manager, you can add the plugin to the configuration as below: ```plaintext Plug 'sourcegraph/sg.nvim', { 'do': 'nvim -l build/init.lua' } " Required for various utilities Plug 'nvim-lua/plenary.nvim' " Required if you want to use some of the search functionality Plug 'nvim-telescope/telescope.nvim' ``` You can source the file and run the command `:PlugInstall` to install the plugin. #### Using Lazy.nvim If you are using Lazy.nvim as the plugin manager, you can add the plugin to the Configuration as below: ```lua return { { "sourcegraph/sg.nvim", dependencies = { "nvim-lua/plenary.nvim", "nvim-telescope/telescope.nvim" }, -- If you have a recent version of lazy.nvim, you don't need to add this! build = "nvim -l build/init.lua", }, } ``` You can source the file and run the command `:Lazy install` to install the plugin. For other installation instructions, you can refer to the [README](https://github.com/sourcegraph/sg.nvim?tab=readme-ov-file#install) of sg.nvim. ### Installing Binaries and Building the Plugin After the plugin is installed, you can move into the building and setup process of the sourcegraph Cody plugin. To install the binaries which are the dependencies of the plugin, you can run the command `:SourcegraphDownloadBinaries` which will force downloading the binaries, making sure that the plugin is properly installed.  To build the plugin, you can simply run the command from within neovim as `:SourcegraphBuild`, this will rebuild the Sourcegraph rust crates and its dependencies (which might have failed during installation). ### Sourcegraph Authentication You need to now authenticate to your Sourcegraph account to use the sourcegraph features such as search and Cody. You can do that by running the command `:SourcegraphLogin` in neovim. This will require two inputs, the sourcegraph endpoint and the access token. If you are using sourcegraph cloud and not a self-hosted sourcegraph, you do not need to change the endpoint, just press enter and move ahead. This will redirect you to the browser for authentication and creating an access token. Log in with your credentials to sourcegraph and copy the access token. This will prompt you back to the neovim interface to provide the access token. Paste the access token there and you will be good to go. ### Health Check Once the plugins are installed then you can check the plugin is correctly downloaded by checking the health with the `:checkhealth sg` command. Below is the health check report on the sourcegraph plugin in neovim. ```plaintext sg: require("sg.health").check() sg.nvim report ~ - Machine: x86_64, sysname: Linux - OK Valid nvim version: table: 0x7ffa0b7bce38 - OK Found `cargo` (cargo 1.70.0) is executable - Use `:SourcegraphDownloadBinaries` to avoid building locally. - OK Found `sg-nvim-agent`: "/home/meet/.local/share/nvim/site/pack/packer/start/sg.nvim/dist/sg-nvim-agent" - OK Found `node` (config.node_executable) is executable. Version: '20.10.0' - OK Found `cody-agent`: /home/meet/.local/share/nvim/site/pack/packer/start/sg.nvim/dist/cody-agent.js - OK Authentication setup correctly - OK endpoint set to: https://sourcegraph.com - OK Found correct binary versions: "1.0.5" = "1.0.5" - OK Sourcegraph Connection info: { access_token_set = true, endpoint = "https://sourcegraph.com", sg_nvim_version = "1.0.5", sourcegraph_version = { build = "256174_2023-12-30_5.2-dbb20677711c", product = "256174_2023-12-30_5.2-dbb20677711c" } } - To manage your Cody Account, navigate to: https://sourcegraph.com/cody/manage - OK Cody Account Information: { chat_limit = 20, chat_usage = 53, code_limit = 500, code_usage = 0, cody_pro_enabled = false, username = "Mr-Destructive" } - OK sg.nvim is ready to run ``` At this point, the sourcegraph plugin is ready to be used. However, we need to set up the plugin in neovim with the default configurations. ### Configuration In your lua setup files, you can set the plugin like this: ```lua require("sg").setup() ``` Source the lua file and restart neovim, this should properly make sourcegraph commands available in the editor. After these steps, Cody is right into neovim. ## Usage To use the plugin, there are multiple commands available within the editor, the complete list of them is given below: ```plaintext SourcegraphBuild SourcegraphClear SourcegraphDownloadBinaries SourcegraphInfo SourcegraphLink SourcegraphLogin SourcegraphSearch CodyAsk CodyChat CodyDo CodyRestart CodyTask CodyTaskAccept CodyTaskNext CodyTaskPrev CodyTaskView CodyToggle ``` You can get more info about these commands with the `help :commandname` command. The command are however self explanatory and simple to use. ### Cody Ask You can quickly parse a prompt as a string to the `:CodyAsk` command as `:CodyAsk "what is neovim by the way?"` and you will get Cody's response in the side vertical split. ### Cody Chat You can start Cody chats from neovim with the command `:CodyChat`. This will open a vertical split with the Cody chat interface, the bottom split has the user input prompt and the upper window will have the generated Cody response. You can enter the prompt in the bottom buffer get into normal mode and hit enter to send the prompt to generate a response from Cody.  ## Conclusion Sourcegraph Cody is a great tool for getting quick solutions to trivial as well as specific problems to the current file/package or project. The context parsing of Cody makes it valuable for developers to ask for specific questions and it answers them really in a straightforward way, without the developer needing to parse the context for the prompt. Thank you for reading. Happy Coding :) https://www.meetgor.com/posts/2023-review Introduction 2023, what a year! If you're a developer and haven't heard the word 'AI' a million times, where have you been? With the introduction of Chat GPT in Sun, 31 Dec 2023 00:00:00 UTC ## Introduction 2023, what a year! If you're a developer and haven't heard the word 'AI' a million times, where have you been? With the introduction of Chat GPT in December 2022, the start of 2023 was buzzing with Open AI's flagship product. I also did a [2022 review](https://www.meetgor.com/2022-review/). The year 2023 was a pivotal year for me to gain experience in solving and delivering real-world problems to solutions. I went from a code-first approach to a code-last approach for solving a problem. The year was not ideal for freshers, as from the end of 2022 till 2023 there was a huge round of layoffs in big tech companies. The competition was too much for freshers and the AI hype added salt to the injury. I won't say that 2023 was a dark year, but a year that changed the trajectory of tech forever. I am grateful that I witnessed the introduction of AI in all domains, especially in tech. I think this year and the coming years will be exciting if you are a passionate developer, because I feel it is like an industrial revolution but in software development, there are LLMs, tools, and services that help developers boost their productivity and avoid laborious and boilerplate tasks. In 2023 * Continued Solar Labs Internship (Backend) * Contributed to Steampipe and MindsDB * Joined Docsumo as a Python Intern * Wrote 10 articles on Golang * Finalist in Smart India Hackathon Crossing Checklists of 2023 from 2022: * <s>Golang Deep Dive with Open Contributions and Projects</s> * <s>Machine Learning</s> * Kubernetes * Rust * Kafka ## Focus of 2023 The year was mostly spent on internship tasks and writing go code for open-source projects. I have created over 6 SDKs in Golang for various APIs, Services, and Products. I had worked on APIs and specific tasks related to the products of the company I interned at. ### The Solar Labs Internship Carrying forward where I left off in 2022, I interned from July 2022 to January 2023 at [The Solar Labs](https://arka360.com/) (now Arka 360) as a Backend Intern. The initial offer was for a 6-month remote internship. However, they were generous and offered an extension of 4 months i.e. till April 2023. So, I continued the internship as a backend intern from July 2022 till April 2023 i.e. 10 months. In this period I went from a novice developer to a somewhat organized and thoughtful developer. I learned a lot of things including: * Documenting APIs (keeping the structure clean and consistent) * Collaborating with Peers for integration * Deeply understanding of requirements and clarifying the scope I was grateful to my mentors and seniors at this company for walking me through their experiences and guidance for the entire duration. Their patience with my silly doubts and mistakes gave me valuable feedback and helped me whenever I reached out. It was a pleasant experience working and learning with them. ### Job Hunting I had taken a break of 20 days from coding, from April 27 to May 10th. This was a break when I traveled to Uttarakhand, did trekked at Kedarnath and Tungnath. It was a refreshing trip and one of the most memorable trips I will ever have with my family. Vacations were over and it was time to find other internships or jobs. I was in the shadows till now, as I had the internship from January to April. I didn't notice how bad the market was for getting a job, layoffs were happening right and left. But this is where I hit reality and was truly exposed to the tech market of 2023. #### 2022 vs 2023 Tech Job Market I got The Solar Labs Internship last year in 2022, I had just applied to 4-5 different openings, and in the 3rd application itself, I was invited to the interviews. Luckily I cleared the 2 rounds and got the internship. But this was 2023, the year that will be remembered as a drought of tech hiring. I started applying in June 2023, since I had semester exams in May. I applied to 5-10 openings, I knew it would be a bit hard since the layoffs were happening and the process might be a bit challenging. No response from those applied to another 10-15 openings, but no response, and it continued. Some responses were coming like I knew I had applied to a position where experience was required but I was a fresher so I was rejected. Weeks passed, months were passing, it was almost July end and on-campus placement started in college a ray of hope, but I didn't have much expectations from it. I was applying casually and getting a feel of the selection process. Something struck me in July and I remembered a company reaching out to me in October 2022, but I had to reject it since I was already in the internship, I however said I might look into next year. But things hit hard when I approached them in July 2023 requesting an opening of a junior developer, but the positions were closed. This is the point where everything starts to fall apart, I start losing my confidence and burning in the self-doubts. I was still applying to internships and jobs on campus as well as off-campus drives. #### Someone notices my skills Months passed, it was almost September 2023, and the ray of hope finally shone with a phone call for interview selection from Docsumo. Scheduled the interviews one after the other and took a week to complete the process, I was selected for a 2-month internship. I was not sure whether to be happy or sad, definitely it was a great startup but just for 2 months. It would feel as if I was just getting started, warm up, and got disconnected. But I took that offer to join from November to January. It was a heck of a ride to land an internship in 2023, I think I was lucky I got it here, it could have been worse. #### Job Market narrative I would like to emphasize the approach for getting a job/internship in 2024: * Be honest and clear about your goals and skills * Learn the fundamentals of the thing you are looking to get hired for * Learn in public * Be proactive while telling your introduction * Keep up with the latest trends and technologies It would be over exaggeration no one can master all of them, but just be aware of what are you headed towards. It is a marathon, not a sprint, take it slowly and allow yourself to build a strong foundation, it will work in the long run. ### Docsumo Internship I was not as excited about the Solar Labs internship as I was about the Docsumo one, as it was a hybrid. I felt a bit disconnected and not able to clearly express myself while working remotely sometimes. So, I felt Docsumo would be a good way to make myself comfortable speaking with seniors and collaborating. I joined in November, had a brief onboarding process, and was introduced nicely with the product itself. In the next week, I was given the task that I will be working over the few months of my stay at Docsumo. I have almost completed over a month at Docsumo and have had a great experience so far. It has changed my naive mindset while approaching a problem with a solution first and then code. It has helped me research a particular thing more deeply and use existing and well-tested solutions rather than reinventing the wheel. Still, there is a lot to learn and improve, will try my best to get good memories till I am here. Whatever happens, even for a short duration, they have given me hope. I am and will be grateful for the opportunity they have given me a driveling to work here, ### Articles Written I mostly wrote about Golang and HTMX(Django) this year with a total of 14 posts: * [100 days of Go Series](https://www.meetgor.com/series/100-days-of-golang) * [HTMX + Django Chat Application](https://dev.to/mr_destructive/creting-a-chat-application-with-django-and-htmx-1095) * Dev. to x Refine hackathon: [Utility Studio](https://dev.to/mr_destructive/utility-studio-quick-ai-powered-utilities-335j) * Hashnode x Grafbase Hackathon: [Grafbase MindsDB](https://techstructiveblog.hashnode.dev/mindsdb-grafbase-graphql-api) ### Open Source Contributions While I was applying for jobs and internships, I was actively contributing to open-source projects like Steampipe and mindsdb. I had been contributing in bits and pieces to Turbot's steampipe CLI for over a year. I got introduced to MindsDB in mid-2023 and started contributing there as well. My primary languages for the contributions were Go and Python. I picked Golang last year and have been loving it since the beginning. I created over 6 Go SDKs for various APIs, Products, and Services like: * [Turso Go](https://github.com/Mr-Destructive/turso-go) * [PaLM2 API](https://github.com/Mr-Destructive/palm) * [Substack](https://github.com/Mr-Destructive/substack-go) * [MindsDB](https://github.com/Mr-Destructive/mindsdb_go_sdk) * [HuggingChat](https://github.com/Mr-Destructive/hugging-chat-go) Contributed to MindsDB Python SDK, Steampipe CLI, and LibSQL(Turso) shell, with the following sets of Pull requests: * [Turbot Steampipe CLI](https://github.com/turbot/steampipe/pulls?q=is%3Apr+is%3Aclosed+author%3Amr-destructive+) and [cohere ai plugin](https://github.com/Mr-Destructive/steampipe-plugin-cohereai) * [MindsDB Handlers](https://github.com/mindsdb/mindsdb/pulls?q=is%3Apr+is%3Aclosed+author%3Amr-destructive+) * [Turso CLI](https://github.com/search?q=org%3Atursodatabase++mr-destructive&type=pullrequests) I loved working with these communities and look forward to contributing and learning more from these interactions. I am excited to check out flowpipe from Turbot, add a couple of plugins on the Steampipe hub, add a few handlers to mindsdb, and integrate Turso db with some golang and Python projects I have in mind. ### Non Technical Things Apart from tech, there are other things as well I do in my free time like reading science fiction, and fantasy novels, writing short stories or novellas, travelling, and gaming. This year, I was able to balance many non-technical things as well like: * Read 7 books (all fiction) -&gt; [Goodreads Profile](https://www.goodreads.com/user_challenges/44008346) * Started to write 2 novellas * Trekked to the highest Shiva temple, [Tungnath](https://en.wikipedia.org/wiki/Tungnath) * Maintained a daily journal throughout the year (at least 5 lines, digital) ## Goals for 2024 The aim is still to be curious and actively adapt myself to learn new technologies with an open mindset. Some high-level goals for my tech journey moving into 2024 will look something like this: * Build a consistent blog writing routine * Complete the 100-days-of-Golang Series * Deep dive into Kubernetes and Micro-services * Write a technical book * Continue contributing to open source more actively in communities I am focusing on writing technical blogs/documentation as it enhances my learning and increases my understanding on a deeper level. So, I will look forward to bringing back the number-crunching blog posts every week. Also, 2024 will be the year I will graduate from the university, so June 2024 is when I'll complete my final exams, and probably the convocation will happen in August. After that, I am a B.Tech I.T. graduate. Time went quickly! ## Conclusion At the beginning of 2023, I was shy and under-confident, I didn't speak up clearly, and I made random side projects(I still do). But at the end of 2023, I had improved my communication skills and felt a lot more confident and expressive than I used to be. I still make random side projects but they are more refined and meaningful in what I want and achieve as a specific thing I want to make. 2023 was a bitter sweat year in terms of expectations and results, but it had a significant impact on my growth as a stronger and better developer. Hopefully, 2024 will be a better and more challenging one. I always remember this quote when I feel challenged > Smooth seas never made skilled sailors With 2023 finally wrapping up, It is time to clear up our bad memories and get excited for the rays of hope and new beginnings in 2024. I am heart-filled thank you for reading through this and I hope 2024 will be a great and happy year for you. Happy New Year and as always, Happy Coding :) https://www.meetgor.com/posts/libsql-python Introduction LibSQL is an **Open Contribution** fork of SQLite. Open Contribution means that it allows suggestions and contributions from the community as oppos Thu, 30 Nov 2023 00:00:00 UTC ## Introduction LibSQL is an **Open Contribution** fork of SQLite. Open Contribution means that it allows suggestions and contributions from the community as opposed to SQLite which is open source but doesn't accept community contributions. ## Installation of LibSQL Client(s) There are two libraries for LibSQL to interact with Python, the [libsql-client](https://github.com/libsql/libsql-client-py/) and the [libsql-experimental-python](https://github.com/libsql/libsql-experimental-python). The former is the recommended client as it is stable, whereas the latter is in development and has the latest features from the libsql database engine, however, it is compatible with the [sqlite](https://docs.python.org/3/library/sqlite3.html) module. There are differences in how to connect and fetch responses in each of them, but if you want to get started quickly you can safely turn with the `libsql-client` as it is recommended from the official documentation. If you want to connect an existing SQLite database to libsql, you can turn up the `libsql-experimental-pyhon` package. We will explore both of them in this article. Let's dive straight into the installation of the package. ## LibSQL Client So, Libsql-client is the Python package provided by Turso as a Python client to interact with a libsql/sqlite database. ### Installation The installation of a Python package is as simple as a pip install. ```bash pip install libsql-client ``` ### Connecting to the database Connecting to a libsql database is as simple as SQLite for the local database file. However, if you have a libsql database over the edge(Turso), you can use the API provided by SQLD to connect to that database. #### Connecting Local database file To connect to a simple libsql database file, you can either create a file as `mydb.db` or just move into the code straight away. ```python import libsql_client client = libsql_client.create_client_sync("file:temp.db") result = client.execute("SELECT 1;") for row in result.rows: print(row) client.close() ``` OR ```python import libsql_client with libsql_client.create_client_sync("file:temp.db") as client: result = client.execute("SELECT 1;") for row in result.rows: print(row) ``` So, in the above code, the `client` object is a client of the libsql database. We use the [create\_client\_sync](https://libsql.org/libsql-client-py/reference.html#create_client_sync) method that accepts a few parameters `url` i.e. the URL of the database we want to connect to, and the `auth_token` which we will see when we want to connect to a database on the edge. There is also [create\_client](https://libsql.org/libsql-client-py/reference.html#create_client) method which is an async method. You can turn up to this method if you are looking for async connections. We have used `file:temp.db` i.e. to connect a local file in the current directory named `temp.db` . This method will return a [Client](https://libsql.org/libsql-client-py/reference.html#Client) object in this case a [ClientSync](https://libsql.org/libsql-client-py/reference.html#ClientSync) object which is a wrapper around the `Client` object. We will be using the methods available for the Client class later while querying in detail. You can see that we have used the [execute](https://libsql.org/libsql-client-py/reference.html#ClientSync.execute) method from the client object. #### Connecting to a cloud(turso) database To connect to a libsql database on the edge i.e. turso you need to specify the auth token(JWT) in the `create_client` method. You can create the token for accessing the database with the ```bash turso db tokens create mydb-name ``` This command will create a `JWT_TOKEN` , store this token securely and this will be used to access the database which is hosted on the Turso cloud. ```bash export JWT_TOKEN="YOURTOKEN" OR ## Save it in the .env file ## JWT_TOKEN=YOURTOKEN ``` This will store the token in the environment variable and can be later used from the local environment. ```python import libsql_client with libsql_client.create_client_sync( "libsql://dbname-orgname.turso.io", auth_token="secret.something.secret" ) as client: result = client.execute("SELECT 1;") for row in result.rows: print(row) ``` The `auth_token` is the JWT Token that we created in the previous step, you can load the token as an Environment variable with the following code: ```python import os JWT_TOKEN = os.environ.get("JWT_TOKEN") ## OR # pip install python-dotenv from dotenv import load_dotenv load_dotenv() JWT_TOKEN = os.environ.get("JWT_TOKEN") ``` This token can be used to authenticate the connection to the Turso database. The rest of the connection code remains as it is. ### Running Queries and Fetching results Now, that we have the connection, we can move into querying the database through the client connection. ## LibSQL-Client (SQLite compatible) ### Installation The installation of the [libsql-experimental-python](https://badge.fury.io/py/libsql-experimental) package is simple with the pip install command with the name of the package. ```bash pip install libsql-experimental ``` Now, we can move into connecting to the database with Python script. ### Connecting to the database Connecting to a libsql database is as simple as SQLite for the local database file. ### Syncing with a remote database ### Running Queries and Fetching results https://www.meetgor.com/posts/golang-web-get-method Introduction In this section of the series, we will be exploring how to send a HTTP request in golang. We will be understanding how to send a basic GET request, Sat, 28 Oct 2023 00:00:00 UTC ## Introduction In this section of the series, we will be exploring how to send a `GET` HTTP request in golang. We will be understanding how to send a basic GET request, create an HTTP request and customize the client, add headers, read the response body, etc in the following sections of this post. ## What is a GET method? A [GET](https://en.wikipedia.org/wiki/HTTP#Request_methods) method in the context of an HTTP request is an action that is used to obtain data/resources. The `GET` method is used in a web application to get a resource like an HTML page, image, video, media, etc. Some common usecases of the `GET` method are: - Loading a webpage - Getting an image, file or other resource - API requests to retrieve data - Search queries sending filters and parameters ## Basic GET Request To use the HTTP method `GET` in golang, the [net/http](https://pkg.go.dev/net/http) package has a [Get](https://pkg.go.dev/net/http#Get) method. This method simply takes in a URL as a string and returns the [response](https://pkg.go.dev/net/http#Response) or an error. Let's look at how to send a basic HTTP GET request in golang. ```go // web/methods/get/main.go package main import ( "fmt" "net/http" ) func main() { reqURL := "https://www.google.com" resp, err := http.Get(reqURL) if err != nil { panic(err) } fmt.Println(resp) fmt.Println(resp.Status) fmt.Println("Status Code:", resp.StatusCode) } ``` ``` $ go run main.go &{200 OK 200 HTTP/2.0 2 0 map[Alt-Svc:[h3=":443"; ma=2592000,h3-29=":443"; ma=2592000] Cache-Control:[private, max-age=0] Content-Security-Policy-Report-Only:[object-src 'none';base-uri 'self';script-src 'nonce-pdz_s8Gr0owwMbX8I9qNEQ' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp] Content-Type:[text/html; charset=ISO-8859-1] Date:[Fri, 27 Oct 2023 09:37:04 GMT] Expires:[-1] P3p:[CP="This is not a P3P policy! See g.co/p3phelp for more info."] Server:[gws] Set-Cookie:[1P_JAR=2023-10-27-09; expires=Sun, 26-Nov-2023 09:37:04 GMT; path=/; domain=.google.com; Secure AEC=Ackid1Q5FARA_9d7f7znggUdw6DoJA1DBpI17Z0SWxN519Dc64EqmYVHlFg; expires=Wed, 24-Apr-2024 09:37:04 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax NID=511=EToBPqckCVRE7Paehug1PgNBKqe7lFLI9d12xJrGbvP9r8tkFIRWciry3gsy8FZ8OUIK4gE4PD-irgNzg4Y1fVePLdyu0AJdY_HcqL6zQYok-Adn-y5TDPmMCNuDnrouBfoxtqVjYY_4RFOe8jalkYto5fQAwzWnNJyw8K0avsw; expires=Sat, 27-Apr-2024 09:37:04 GMT; path=/; domain=.google.com; HttpOnly] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[0]] 0xc000197920 -1 [] false true map[] 0xc0000ee000 0xc0000d8420} 200 OK Status Code: 200 ``` In the above code, we have defined a URL string as `reqURL` and used the [Get](https://pkg.go.dev/net/http#Get) method to send a GET request. The `Get` is parsed with the `reqURL` string and the return values are stored as `resp` and `err`. We have added an error check after calling the `Get` method in order to avoid errors later in the code. The `Get` method as seen from the output has returned a `*http.Response` object, we can use the `Status` and `StatusCode` properties to get the status code of the response. In this case, the status code of the response was `200`. The response `resp` is an object of type `http.Response` i.e. it has fields like `Body`, `StatusCode`, `Headers`, `Proto`, etc. We can get each individual field from the `resp` object. We will later look into how to read the `Body` field from the response, it is not directly read as a string nor it is stored in other forms, rather it is streamed from the requested URL. ## Creating a GET request We can even construct a GET request using the [NewRequest](https://pkg.go.dev/net/http#NewRequest) method. This is a low-level way of creating a `GET` request. We mention the `method`, `URL`, and the `body`, in the case of `GET` request, there is nobody. So, the `NewRequest` is a general way of creating a `http` request. ```go // web/methods/get/newreq.go package main import ( "fmt" "net/http" ) func main() { reqURL := "https://www.google.com" req, err := http.NewRequest(http.MethodGet, reqURL, nil) if err != nil { panic(err) } resp, err := http.DefaultClient.Do(req) if err != nil { panic(err) } fmt.Println(resp) } ``` As we can see, we construct a `GET` request using the `NewRequest` method and then use the [Do](https://pkg.go.dev/net/http#Client.Do) method to send the request to the server. The [http.DefaultClient](https://pkg.go.dev/net/http#DefaultClient) is used as a client to send the request, if we want to customize this we can create a new instance object of [http.Client](https://pkg.go.dev/net/http#Client) and use it to send requests. We will be taking a look at clients in another part of this series when we want to persist a connection or avoid connecting multiple times to the same application/URL. For now, we will go ahead with the DefaultClient. This will trigger the request, in this case, a `GET` request to the specified URL in the `reqURL` string. The `Do` method returns either a `http.Response` or an `error` just like the `Get` method did. ## Reading the Response Body We saw some different ways to send a `GET` request, now the below example will demonstrate how to read the body of the response. The response body is read from a buffer rather than loading the entire response into memory. It makes it flexible to parse the data efficiently and as per the needs. We will see how we use the [io](https://pkg.go.dev/io) package's [ReadAll](https://pkg.go.dev/io#ReadAll) method can be used to read from the response buffer. ```go // web/methods/get/body.go package main import ( "fmt" "io" "net/http" ) func main() { reqURL := "https://httpbin.org/html" resp, err := http.Get(reqURL) if err != nil { panic(err) } // close the body object before returning the function // this is to avoid the memory leak defer resp.Body.Close() // stream the data from the response body only once // it is not buffered in the memory body, err := io.ReadAll(resp.Body) if err != nil { panic(err) } fmt.Println(string(body)) } ``` In the above example, we are trying to get the body from the response to the request sent at [`https://httpbin.org/html`](https://httpbin.org/html). We have used the simple `Get` method instead of `NewRequest` and `Do` for simplicity. The response is stored in `resp`, we also have added `defer resp.Body.Close()` which is to say that we will close the body reader object when the function is returned/closed. So, this means that the `Body` is not readily available data, we need to obtain/stream the data from the server. We have to receive the body in bytes as a tcp request, the body is streamed in a buffer. The response body is streamed from the server, which means that it's not immediately available as a complete data set. We read the body in bytes as it arrives over the network, and it's stored in a buffer, which allows us to process the data efficiently. ### Reading Body in bytes We can even read the body in bytes i.e. by reading a chunk of the buffer at a time. We can use the [bytes.Buffer](https://pkg.go.dev/bytes#Buffer) container object to store the body. Then we can create a slice of bytes as `[]bytes` of a certain size and read the body into the chunk. By writing the chunk into the buffer, we get the entire body from the response. ```go // web/methods/get/body.go package main import ( "bytes" "fmt" "net/http" ) func main() { reqURL := "https://httpbin.org/html" resp, err := http.Get(reqURL) if err != nil { panic(err) } defer resp.Body.Close() // create a empty buffer buf := new(bytes.Buffer) // create a chunk buffer of a fixed size chunk := make([]byte, 1024) for { // Read into buffer n, err := resp.Body.Read(chunk) if err != nil { break } // append the chunk to the buffer buf.Write(chunk[:n]) fmt.Printf("%s ", chunk[:n]) } // entire body stored in bytes fmt.Println(buf.String()) } ``` In the above example, the body is read chunkwise buffers and obtained as a slice of bytes. We define the request as a `GET` request to the [`httpbin.org/html`](http://httpbin.org/html). We create a new Buffer as a slice of bytes with [bytes.Buffer](https://pkg.go.dev/bytes#Buffer), then we define chunk as a container to stream the response body with a particular size. We have taken `1024` bytes as the size of the chunk. Then inside an infinite for loop, we read the body as `n, err :=` [`resp.Body.Read`](http://resp.Body.Read)`(chunk)`. The code will read the body into the chunk(slice of byte) and the return value will be the size of the bytes read or an error. Then we check if there is no error, and if there is an error, we break the loop indicating we have completed reading the entire body or something went wrong. Then we write the chunk into the buffer that we allocated earlier as `buf`. This is a slice of bytes, we are basically populating the buffer with more slices of bytes. The entire body is then stored in the buffer as a slice of bytes. So, we have to cast it into a string to see the contents. So, this is how we can read the contents of a body in a response in chunks. ### Parsing the JSON body with structs If we have the structure of the response body already decided, then we can define a struct for the response body and then we can [Unmarshal](https://doc.akka.io/docs/akka-http/current/common/unmarshalling.html#unmarshalling:~:text=Unmarshalling,type%20T.) / deserialize/unpickle. This means we can convert the bytes representation of the data into a Golang-specific structure which is called a high-level representation of the data. We can parse the JSON body into a defined struct using [Unmarshal](https://pkg.go.dev/encoding/json#Unmarshal) or [Decode](https://pkg.go.dev/encoding/json#Decoder.Decode) methods in the [json](https://pkg.go.dev/encoding/json) package. Let's look at both the methods. #### Using Unmarshal The `Unmarshal` method takes in two parameters i.e. the body in bytes and the reference of the object that we want to unmarshal into. The method returns an error if there is a discrepancy in the returned JSON or the structure defined it is unable to deserialize the JSON object into the defined structure. ```go package main import ( "encoding/json" "fmt" "io" "net/http" ) type Product struct { ID int `json:"id"` Title string `json:"title"` Description string `json:"description"` Price float64 `json:"price"` DiscountPercentage float64 `json:"discountPercentage"` Rating float64 `json:"rating"` Stock int `json:"stock"` Brand string `json:"brand"` Category string `json:"category"` Thumbnail string `json:"thumbnail,omitempty"` Images []string `json:"-"` } func main() { reqURL := "https://dummyjson.com/products/1" resp, err := http.Get(reqURL) if err != nil { panic(err) } defer resp.Body.Close() body, err := io.ReadAll(resp.Body) if err != nil { panic(err) } var data Product if err := json.Unmarshal(body, &data); err != nil { panic(err) } fmt.Println(data) fmt.Println(data.Title) } ``` ```bash $ go run main.go {1 iPhone 9 An apple mobile which is nothing like apple 549 12.96 4.69 94 Apple smartphones https://cdn.dummyjson.com/product-images/1/thumbnail.jpg []} iPhone 9 ``` In the above example, we have defined a structure called Product with fields such as `ID`, `Title`, `Description`, etc. We use the JSON tag to specify how each field should be encoded to or decoded from JSON. These tags guide the Golang JSON encoders and decoders to correctly map JSON data to struct fields and vice versa. The `omitempty` option in a struct tag instructs the encoder to omit the field from the JSON output if the field's value is the zero value for its type (e.g., 0 for integers, "" for strings, nil for pointers, slices, and maps). This is useful for reducing the size of the JSON output by excluding empty or default-valued fields. Conversely, the `-` option in a struct tag tells the encoder and decoder to completely ignore the field. It will not be included in encoded JSON, nor will it be populated when decoding JSON into a struct. This is particularly useful for excluding fields that are meant for internal use only and should not be exposed through JSON. Therefore, `omitempty` is used to control the inclusion of fields in the JSON output based on their values, while `-` is used to exclude fields from both encoding and decoding from JSON. We send the `GET` request to the api `https://dummyjson.com/products/1`. The response from the request is read into a slice of bytes with [io.ReadAll](https://pkg.go.dev/io#ReadAll) that takes in a [io.Reader](https://pkg.go.dev/io#Reader) object in this case it is the `resp.Body` and it returns a slice of byte and error if any issue arises while reading in the body. Further, we can use the [Unmarshal](https://pkg.go.dev/encoding/json#Unmarshal) method to parse the slice of body `body` into the struct `Product` with the variable `data`, the reference to `&data` indicates that the method will directly mutate/change this variable and populate the object with the fields from the body. So in a gist, to convert the JSON body into a golang native structure with `Unmarshal` with the following steps: - Read the body into a slice of bytes using `io.ReadAll` - Create an object of the struct - Pass the body as a slice of bytes and the reference of that object (struct instance) into the Unmarshal method - Access the object with the fields in the struct In the output response, we can see the object is populated with the fields from the body. The `Title` field is accessed using the `data.Title` as we do with a normal golang struct. The `Images` field is not populated because we have always ignored/omitted from the json tag with `-`. #### Using Decoder Similar to the `Unmarshal` we can use the [Decoder](https://pkg.go.dev/encoding/json#Decoder) to parse the body into a struct. However, the parameters it takes are a bit different and it is a two-step process. We first create a [Decoder](https://pkg.go.dev/encoding/json#Decoder) object using the [NewDecoder](https://pkg.go.dev/encoding/json#NewDecoder) method, which takes in a `io.Reader` object, luckily the body from the response is already in that structure, so we can directly pass that `resp.Body` into the `NewDecoder` method. The second step is to decode the data into the object, here as well, we need to create the object of the struct and parse the reference to the object to the [Decode](https://pkg.go.dev/encoding/json#Decoder.Decode) method. The `Decode` method converts the bytes parsed in the `resp.Body` from the `Decoder` object and populates the fields of the object provided in the reference struct. So the steps for deserializing the json object into the struct with the decode method are: - Create a decoder with `NewDecoder` method and pass the `resp.Body` as the parameter which is an `io.Reader` object - Create an object of the struct - Decode the body into that object using the `decoder.Decode` method - Access the object with the fields in the struct ```go package main import ( "encoding/json" "fmt" "net/http" ) type Product struct { ID int `json:"id"` Title string `json:"title"` Description string `json:"description"` Price float64 `json:"price"` DiscountPercentage float64 `json:"discountPercentage"` Rating float64 `json:"rating"` Stock int `json:"stock"` Brand string `json:"brand"` Category string `json:"category"` Thumbnail string `json:"thumbnail,omitempty"` Images []string `json:"-"` } func main() { reqURL := "https://dummyjson.com/products/1" resp, err := http.Get(reqURL) if err != nil { panic(err) } defer resp.Body.Close() var data Product decoder := json.NewDecoder(resp.Body) err = decoder.Decode(&data) if err != nil { panic(err) } fmt.Println(data) fmt.Println(data.Title) } ``` ```bash $ go run main.go {1 iPhone 9 An apple mobile which is nothing like apple 549 12.96 4.69 94 Apple smartphones https://cdn.dummyjson.com/product-images/1/thumbnail.jpg []} iPhone 9 ``` We have first defined the struct `Product` with the `json:"id"` tag. As explained earlier, we have used the json tags to identify the fields from the json data to the structures while encoding and decoding. In the above example, we have sent a `GET` request to the api endpoint `https://dummyjson.com/products/1`, and we have created a new decoder with the `NewDecoder` method with the `resp.Body` as the parameter. The `data` is created as a `Product` instance. The reference to `data` is parsed to the `Decode` method from the `decoder` instance as `&data`. This method will either return `nil` or an `error`. Thereafter, we can check for errors and then only access the data object with its populated fields from the response body. There is a certain difference between the `Unmarshal` and `Decode` methods. The difference is just a slight performance improvement in the `NewDecoder` and `Decode` methods. Though it is not significant, having a little info about it might be handy in your use case. Read here for more info : [To Unmarshal or Decode](https://dev.to/jpoly1219/to-unmarshal-or-to-decode-json-processing-in-go-explained-5870) ## Adding Headers to a GET Request We can even add headers before sending a `GET` request to a URL. By creating a `Request` object with the `NewRequest` method and adding a [Header](https://pkg.go.dev/net/http#Header) with the [Add](https://pkg.go.dev/net/http#Header.Add) method. The `Add` method will take in two parameters i.e. the key of the header, and the value of the header both as strings. ```go // web/methods/get/header.go package main import ( "fmt" "io" "net/http" ) func main() { req, err := http.NewRequest(http.MethodGet, "https://api.github.com/users/mr-destructive", nil) if err != nil { panic(err) } req.Header.Add("Authorization", "token YOUR_TOKEN") resp, err := http.DefaultClient.Do(req) if err != nil { panic(err) } defer resp.Body.Close() body, err := io.ReadAll(resp.Body) if err != nil { panic(err) } fmt.Println(string(body)) } ``` ``` $ go run web/methods/get/header.go {"message":"Bad credentials","documentation_url":"https://docs.github.com/rest"} ``` In the above example, we have created a `GET` request to the [`https://api.github.com/users/mr-destructive`](https://api.github.com/users/mr-destructive) the last portion is the username, it could be any valid username. The request is to the GitHub API, so it might require API Key/Tokens in the headers, however, if there are certain endpoints that do not require Authorization headers might work just fine. So, the above code will give `401` error indicating the request has wrong or invalid credentials, if we remove the header, the request will work fine. This is just an example, but headers are useful in working with APIs. Without adding the header: ``` $ go run web/methods/get/header.go {"login":"Mr-Destructive","id":40317114,"node_id":"MDQ6VXNlcjQwMzE3MTE0","avatar_url":"https://avatars.githubusercontent.com/u/40317114?v=4","gravatar_id":"","url":"https://api.github.com/users/Mr-Destructive", ... "updated_at":"2023-10-10T17:57:22Z"} ``` That's it from the 33rd part of the series, all the source code for the examples are linked in the GitHub on the [100 days of Golang](https://github.com/Mr-Destructive/100-days-of-golang/tree/main/web/methods/get/) repository. [100-days-of-golang](https://github.com/Mr-Destructive/100-days-of-golang) ## References - [To Unmarshal or Decode](https://dev.to/jpoly1219/to-unmarshal-or-to-decode-json-processing-in-go-explained-5870) - [Golang JSON tutorial](https://drstearns.github.io/tutorials/gojson/) - [Golang OmitEmpty](https://www.sohamkamani.com/golang/omitempty/) ## Conclusion From this article, we explored the `GET` HTTP method in golang. By using a few examples for creating a get request, adding headers, reading response body, the basic use cases were demonstrated. Hopefully, you found this section helpful, if you have any comments or feedback, please let me know in the comments section or on my social handles. Thank you for reading. Happy Coding :) https://www.meetgor.com/posts/golang-web-url-parsing Introduction We have done around 32 posts on the fundamental concepts in golang, With that basic foundation, I'd like to start with the new section of this seri Tue, 05 Sep 2023 00:00:00 UTC ## Introduction We have done around 32 posts on the fundamental concepts in golang, With that basic foundation, I'd like to start with the new section of this series which will be a major one as `web-development`. This section will have nearly 40-50 posts, this will cover the fundamental concepts for web development like APIs, Database integrations, Authentication and Authorizations, Web applications, static sites, etc. ## What is a URL? A URL is a Uniform Resource Locator. It is a string of characters that identifies a resource on the Internet. URLs are the building blocks of the web, allowing us to access websites, documents, and data with just a click. URLs are all over the place, if we want to build a strong foundation in web development, it's quite important to understand what URLs actually mean and what can they store. A URL looks something like this: ``` [scheme:][//[userinfo@]host][/]path[?query][#fragment] ``` Not all the URLs are like this, majority of the URLs that the common user sees are simply the ones with the `scheme`, `host`, and `paths`. However other components are equally important and are vital in the exchanging of information over the network. - The `scheme` is the protocol used for accessing the resource like `http`, `https`, `ftp`, etc. - The `userinfo` is the username and password used to access the resource. - The `host` is the domain name of the resource. - The `path` is the path or folder to the resource. - The `query` is the query string of the resource. It is usually a key-value pair as a paramter to access resources. - The `fragment` is used as a reference within the resource. We will see the use cases of most of them throughout this series for example, the `userinfo` is commonly used in accessing databases over the internet/cloud. The query parameters will be used in making dynamic API calls, etc. ## Basic URL Parsing ```go package main import ( "fmt" "net/url" ) func main() { // simple url urlString := "http://www.google.com" parsedUrl, err := url.Parse(urlString) if err != nil { panic(err) } fmt.Println(parsedUrl) } ``` ``` $ go run main.go http://www.google.com ``` So, what is getting parsed here, we gave the URL as a string we get the URL back, the only difference is that instead of the URL being a string, it is now a structure of components. For instance, we want the protocol the host name, the port, etc. from the URL. ``` fmt.Printf("%T ", parsedUrl) // *url.URL ``` The `parsedUrl` is a pointer to a [url.URL](https://pkg.go.dev/net/url#URL) structure. The structure `url.URL` has a lot of components to it like `Scheme`, `Host`, `User`, `Path`, `RawQuery`, etc. We will dive into each of these soon. We could get those specific components ourselves, but that would be a bit tedious and might be even prone to bugs. Let's try to get those components from the URL without any functions, just string manipulation. ```go package main import ( "fmt" ) func main() { urlString := "http://www.google.com" protocol := urlString.split(":")[0] hostName := urlString.split("//")[1] fmt.Println(protocol) fmt.Println(hostName) } ``` This might work for a simple URL, but what if we have more complex URLs which have paths, query parameters, fragments, username, port, etc? This could mess up quickly if we tried to get the parts of the URL ourselves. So, golang has a package called [net/url](https://pkg.go.dev/net/url) explicitly for parsing URLs. ```go package main import ( "fmt" "strings" ) func main() { urlString := []string{"http://www.google.com", "http://www.google.com/about/", "http://www.google.com/about?q=hello", "http://www.google.com:8080/about?q=hello", "http://user:password@example.com:8080/path/to/resource?query=value#fragment", } for _, url := range urlString { hostStr := strings.Split(url, "//")[1] hostName := strings.Split(hostStr, "/")[0] fmt.Println(hostName) } } ``` ``` $ go run main.go www.google.com www.google.com www.google.com www.google.com:8080 user:password@example.com:8080 ``` The above code might work for most of the URLs, but what if we have a more complex URL like the one with `port` or `user`, it doesn't give the thing we want exactly. In the above example, we have created a list of URLs as strings and simply iterated over each of the `url` in the `urlString`. Thereafter, we split the `url` on `//` so we get `https:` and `www.google.com`, if we want the host/domain name, we could simply get the `1` index in the slice since the [strings.Split](https://pkg.go.dev/strings#Split) method returns a slice after splitting the string with the provided separator. The `hostName` could be fetched from the `1` index. However, this time for the 2nd element in the list, we have `https://www.google.com/about`, which would return `www.google.com/about` as the hostname which is not ideal, so we will again have to split this string with `/` and grab the first part i.e. 0th index. The above code would work for `paths` and `query` parameters but if we had ports, and username, and password, it would not work as expected as evident from the last 2 examples in the list. So, now we know the downsides of manually parsing the URLs, we can use the [net/url](https://pkg.go.dev/net/url) package to do it for us. ## Parsing DB URLs Databases have a connection URL or connection string which provides a standard way to connect to a database/database server. The format of the URL is just the `URL` with all the components from the `scheme` to the `path`. The common examples of some database connection URLs might include: ``` # PostgreSQL DB Connection URL/string postgresql://username:password@hostname:port/database_name # MongoDB Connection URL/string mongodb://username:password@hostname:port/database_name ``` The above are examples of the Postgres and MongoDB connection URLs, they have a `protocol` which usually for databases is their short name, the user credentials i.e. `username` and `password`, the `hostname` i.e. the server IP address, the `port` on which the database is running, and finally the path as the `database name`. We can construct a simple snippet in golang to grab all the details from the simple connection URL string with the `net/url` package. ```go package main import ( "fmt" "net/url" ) func main() { //postgres db url dbUrl, err := url.Parse("postgres://admin:pass1234@localhost:5432/mydb") if err != nil { panic(err) } fmt.Println(dbUrl) fmt.Println("Scheme/Protocol = ", dbUrl.Scheme) fmt.Println("User = ", dbUrl.User) //fmt.Println("User = ", dbUrl.User.String()) fmt.Println("Username = ", dbUrl.User.Username()) password, _ := dbUrl.User.Password() fmt.Println("Password = ", password) fmt.Println("Host = ", dbUrl.Host) fmt.Println("HostName = ", dbUrl.Hostname()) fmt.Println("Port = ", dbUrl.Port()) fmt.Println("DB Name = ", dbUrl.Path) } ``` ``` $ go run main.go postgres://postgres:pass1234@localhost:5432/mydb Scheme/Protocol = postgres User = admin:pass1234 Username = admin Password = pass1234 Host = localhost:5432 HostName = localhost Port = 5432 DB Name = mydb ``` In the above code, we have given the string `postgres://admin:pass1234@localhost:5432/mydb`, and we have parsed the URL using the `net/url` package. The result is we have a `parsedUrl` object which has all the components that can be accessed as either fields or methods. Let's break down each field/method we used in the above example: - The `Scheme` is simply a string representing the protocol of the resource(URL). - The `User` is the [UserInfo](https://pkg.go.dev/net/url#Userinfo) object having immutable username and password fields. - The `Username` is the method on [UserInfo](https://pkg.go.dev/net/url#Userinfo.Username) that returns the string representing the username of the URL. - The `Password` is the method on [UserInfo](https://pkg.go.dev/net/url#Userinfo.Password) that returns the string representing the password of the URL. - The `Host` is the field on `URL` as a string representing the host:port of the URL. - The `Hostname` is the method on [URL](https://pkg.go.dev/net/url#URL.Hostname) that returns the string representing the hostname of the URL. - The `Port` is the method on [URL](https://pkg.go.dev/net/url#URL.Port) that returns the string representing the port of the URL. - The `Path` is the field as the string representing the path of the URL. So, this is how we can get almost every detail like `db protocol`, `username`, `password`, `hostname`, `port`, and the `database name` from the database connection string URL. ## Parsing Query Parameters We can even get the query parameters of a URL using the [Query](https://pkg.go.dev/net/url#URL.Query) method on the `URL` object. The `Query` method returns a `map[string][]string` which is to say a map with the key as `string` and the value as a `[]string` slice of string. For example, if the URL is something like `https://google.com/?q=hello`, the `Query` method will return `map[q:[hello]]` which means the key is `q` and the value is a list of strings of which the only element is `hello`. ```go package main import ( "fmt" "net/url" ) func main() { // a complex url with query params urlStr := "http://www.google.com/?q=hello&q=world&lang=en" parsedUrl, err := url.Parse(urlStr) if err != nil { panic(err) } fmt.Println(parsedUrl) fmt.Println(parsedUrl.Query()) for k, v := range parsedUrl.Query() { fmt.Println("KEY:", k) for _, vv := range v { fmt.Println(vv) } } } ``` ``` $ go run main.go http://www.google.com/?q=hello+world&lang=en&q=gopher map[lang:[en] q:[hello world gopher]] KEY: q hello world gopher KEY: lang en ``` We have taken a bit of a complex example that might cover many use cases of the `Query` method. We have a URL as `http://www.google.com/?q=hello&q=world&lang=en`, and the `Query` method returns `map[lang:[en] q:[hello world]]` which means the `q` key has a slice of a string with elements `hello world` and `gopher` and the `lang` key has a value of `en`. Here, the first paramter, `q=hello+world` is basically `hello world` or `hello%20world`, which is to say escaping the space in the URL. We can have multiple values for the same key, which is evident as we have added `q=gopher` at the end of the `URL`, the key `q` has two elements in the slice as `hello world` and `gopher`. The `lang=en` is simply a key as `lang` with the only element as `en` in the slice. We use `&` to separate different query parameters in the URL. ### Checking Values in Query Parameters We can even check the values in the query parameters without requiring the construction of for loops to find a particular value in a key. The [Values](https://pkg.go.dev/net/url#Values) is a type that stores the map as a return value from the `Query` method. It has a few handy methods like: - [Has](https://pkg.go.dev/net/url#Values.Has) to check if the key exists in the map (paramter as key `string` and returns a `bool`). - [Get](https://pkg.go.dev/net/url#Values.Get) to fetch the first value of the given key as a string or if not present then returns an empty string (paramter as key `string` and returns `string`). - [Add](https://pkg.go.dev/net/url#Values.Add) method is used to append the value for a given key (paramter as key `string` and value to be added as `string`). - [Set](https://pkg.go.dev/net/url#Values.Set) method is used to replace the value for a given key if already exists (paramter as key `string` and value as `string`). - [Del](https://pkg.go.dev/net/url#Values.Del) method is used to delete the value for a given key (paramter as key `string`). ```go package main import ( "fmt" "net/url" ) func main() { // a complex url with query params urlStr := "http://www.google.com/?q=hello+world&lang=en&q=gopher" parsedUrl, err := url.Parse(urlStr) if err != nil { panic(err) } fmt.Println(parsedUrl) fmt.Println(parsedUrl.Query()) queryParams := parsedUrl.Query() fmt.Println(queryParams.Get("q")) fmt.Println(queryParams.Has("q")) if queryParams.Has("lang") { fmt.Println(queryParams.Get("lang")) } queryParams.Add("q", "ferris") fmt.Println(queryParams) queryParams.Set("q", "books") fmt.Println(queryParams) queryParams.Del("q") fmt.Println(queryParams) } ``` ``` $ go run main.go http://www.google.com/?q=hello+world&lang=en&q=gopher hello world true en map[lang:[en] q:[hello world gopher ferris]] map[lang:[en] q:[books]] map[lang:[en]] ``` The above code example demonstrates almost all the methods available on the `Values` type. The `Get` method is used to fetch the first value for a given key, so we parse the key as a `string` to the method and it would return a `string`. We checked for the `q` as the key and it returned the first element in the `queryParams` for key `q` as `hello world` from the list `[hello world, gopher]`. The `Has` method takes in the paramter as key as `string` and returns if the key exists in the `queryParams` or not as a bool. The `Add` method, we have used to `Add` a key with a particular value, we added the value `ferris` to the key `q` hence it appended to the list and the list of `queryParams[q]` became `[hello world, gopher, ferris]`. The `Set` method is used to override the existing key with a particular value, here we have set the value `books` to the key `q` and hence the list of `queryParams[q]` becomes `[books]`. We can use the `Del` method to remove the key from the `queryParams`, so we delete `q` from `queryParams`, then the `queryParams` simply has no key as `q` in it. ### Parsing Query Parameters to String Now, that you have manipulated the query parameters, let's say you want to construct back that string representation of the query particular or the URL for it. The [Encode]() method is used to grab the `queryParams` i.e. `Values` object and convert it into the `string` representation of the encoded URL. ```go package main import ( "fmt" "net/url" ) func main() { // a complex url with query params urlStr := "http://www.google.com/?q=hello+world&lang=en&q=gopher" parsedUrl, err := url.Parse(urlStr) if err != nil { panic(err) } queryParams := parsedUrl.Query() queryParams.Add("name", "ferris") q := queryParams.Encode() fmt.Println(q) } ``` ``` $ go run main.go lang=en&name=ferris&q=hello+world&q=gopher ``` So, we can see the `Encode` method has given a query parameter in the form of a URL encoded string. We first grab the query parameters from the `parsedUrl` which is a `URL` object via the `Query` method, we then `Add` the key `name` with a value of `ferris` to the `queryParams`. This is then used to `Encode` the object back to a string representation. This could be useful to construct a query paramter for requesting other websites/APIs. ## Parsing URL object back to String We can even get the `URL` object back to a string representation using the [String](https://pkg.go.dev/net/url#URL.String) method on the `URL` object. The `String` method returns a `string` representation of the URL object. ```go package main import ( "fmt" "net/url" ) func main() { urlStr := "http://www.google.com/?q=hello+world&lang=en&q=gopher" fmt.Println("URL:", urlStr) parsedUrl, err := url.Parse(urlStr) if err != nil { panic(err) } queryParams := parsedUrl.Query() queryParams.Add("name", "ferris") q := queryParams.Encode() fmt.Println(q) parsedUrl.RawQuery = q newUrl := parsedUrl.String() fmt.Println("New URL:", newUrl) } ``` ``` $ go run main.go URL: http://www.google.com/?q=hello+world&lang=en&q=gopher lang=en&name=ferris&q=hello+world&q=gopher New URL: http://www.google.com/?lang=en&name=ferris&q=hello+world&q=gopher ``` In the example above, we parse a URL string into a `URL` object as `parsedUrl`, then we `Add` the key `name` with a value of `ferris` to the `queryParams`. We then `Encode` the `URL` object back to a string representation. But this won't change the `parsedUrl` object we want to change the entire `URL` object. For that, we have overwritten the `RawQuery` field of the `URL` object with the query parameter encoded string as `q`. The `String` method returns a `string` representation of the `URL` object. ## Parsing Fragments The fragment in a URL is usually present in a static website like `#about`, `#contact`, `#blog`, etc. The `Fragment` is a string that is usually a reference to a specific section or anchor point within a web page or resource. When a URL with a fragment is accessed, the web browser or user agent will scroll the page to display the section identified by the fragment. ```go package main import ( "fmt" "net/url" ) func main() { // url with fragments urlStr := "https://pkg.go.dev/math/rand#Norm Float64" parsedUrl, err := url.Parse(urlStr) if err != nil { panic(err) } fmt.Println(parsedUrl) fmt.Println(parsedUrl.Fragment) fmt.Println(parsedUrl.RawFragment) fmt.Println(parsedUrl.EscapedFragment()) } ``` ``` $ go run main.go https://pkg.go.dev/math/rand#Norm Float64 Norm Float64 Norm Float64 Norm%20Float64 ``` The above code is used to fetch the `#Norm Float64` fragment from the URL `https://pkg.go.dev/math/rand#NormFloat64`. We can use the [Fragment](https://pkg.go.dev/net/url#URL) field in the `URL` object to get the fragment text. There is [RawFragment](https://pkg.go.dev/net/url#URL) field that is used to parse the fragment text as it is, without trying to escape any special characters in the URL. The [EscapedFragment](https://pkg.go.dev/net/url#URL.EscapedFragment) is used to parse the fragment text by escaping the characters in the URL. That's it from the 32nd part of the series, all the source code for the examples are linked in the GitHub on the [100 days of Golang](https://github.com/Mr-Destructive/100-days-of-golang/tree/main/web/url-parsing) repository. [100-days-of-golang](https://github.com/Mr-Destructive/100-days-of-golang) ## Conclusion From the first post of the web development section, we covered the fundamentals of URL parsing and got a bit introduced to the `net` package, which will be heavily used for most of the core language's features for working with the web. We covered the concepts for parsing URLs, getting components of URLs from the parsed object, Database connection URL resolving, parsing query parameters, and some other URL-related concepts. Hopefully, you found this section helpful, if you have any comments or feedback, please let me know in the comments section or on my social handles. Thank you for reading. Happy Coding :) https://www.meetgor.com/posts/golang-channels Introduction In this part of the series, we will be continuing with the concurrency features of golang with channels. In the last post, we covered the fundament Mon, 28 Aug 2023 00:00:00 UTC ## Introduction In this part of the series, we will be continuing with the concurrency features of golang with channels. In the last post, we covered the fundamentals of go routines and wait groups. By leveraging those understood concepts, we will explore channels to communicate the data between various go routines. ## What are Channels A golang Channel is like a pipe that lets goroutines communicate. It lets you pass values from one goroutine to another. Channels are typed i.e. you declare them with `chan` keyword followed by the type to be sent and received (e.g. `chan int`). The `chan` type specifies the type of values that will be passed through the channel. We will explore the detailed technicalities soon. Right now, we need to just focus on what problem is channels solving. In the previous article, we worked with go routines and wait groups which allowed us to process tasks asynchronously. However, if we wanted to access the data in between the processes, we would have to tweak the core functionality or might require global variables, however, in real-world applications, the environment is quite constrained. We would require a way to communicate data between those go routines. Channels are made just for that(more than that), but in essence, it solves that exact problem. ```go package main import ( "fmt" ) func main() { ch := make(chan string) defer close(ch) go func() { message := "Hello, Gophers!" ch <- message }() msg := <-ch fmt.Println(msg) } ``` In the above code example, the channel `ch` is created of type `string` and a message is sent to the channel inside a go routine as `ch <- message`, and the message is retrieved from the channel as `<-ch`. ```bash $ go run main.go Hello, Gophers! ``` Channels have two key properties: - Send and receive operations block until both sides are ready(i.e. there is a sender and a receiver for a channel). This allows goroutines to synchronize without explicit locks or condition variables. - Channels are typed, so only values of the specified type can be sent and received. This provides type safety. ```go package main import ( "fmt" ) func main() { ch := make(chan string) go func() { message := "Hello, Gophers!" ch <- message }() fmt.Println(<-ch) fmt.Println(<-ch) } ``` In the same example, if we tried to add the second receiver i.e. `<-ch`, it would result in a deadlock/block forever since there is no second message sent into the channel. Only one value i.e. "Hello Gophers!" was sent as a `message` into the channel, and that was received by the first receiver as `<-ch`, however in the next receiver, there is no sender. ```bash $ go run main.go Hello, Gophers! fatal error: all goroutines are asleep - deadlock! goroutine 1 [chan receive]: main.main() /home/meet/code/100-days-of-golang/scripts/channels/main.go:16 +0x125 exit status 2 ``` To sum up the deadlock concept in unbuffered channels: - The main goroutine is waiting at the second receive operation for a second message that will never arrive (was never sent). - The anonymous goroutine is waiting for someone to read from the channel so it can proceed with sending the second message. ## Buffered Channels In Go, you can create both buffered and unbuffered channels. An unbuffered channel has no capacity to hold data, it relies on immediate communication between the sender and receiver. However, you can create a buffered channel by specifying a capacity when using the make function, like `ch := make(chan int, 5)` will create a channel with a capacity of `5` i.e. it can store a certain number of values without an immediate receiver. A buffered channel allows you to send multiple values to the channel without an immediate receiver, up to its capacity. After that, it will block until the receiver retrieves values. ```go package main import ( "fmt" "sync" ) func main() { buffchan := make(chan int, 2) wg := sync.WaitGroup{} wg.Add(2) for i := 1; i <= 2; i++ { go func(n int) { buffchan <- n wg.Done() }(i) } wg.Wait() close(buffchan) for c := range buffchan { fmt.Println(c) } } ``` ```bash $ go run channels.go 1 2 $ go run channels.go 2 1 ``` In this code snippet, we create a buffered channel ch with a capacity of 2. We send two values to the channel, and even though there's no immediate receiver, the code doesn't block. If we were to send a third value, it would lead to a deadlock because there is no receiver to free up space in the buffer. ## Closing Channels Closing a channel is important to signal to the receiver that no more data will be sent. It's achieved using the built-in close function. After closing a channel, any further attempts to send data will result in a panic. On the receiving side, if a channel is closed and there's no more data to receive, the receive operation will yield the zero value for the channel's type. ```go package main import ( "fmt" ) func main() { ch := make(chan int) go func() { for i := 1; i <= 5; i++ { ch <- i } close(ch) }() for num := range ch { fmt.Println("Received:", num) } } ``` In this example, a goroutine sends numbers to the channel and then closes it. The main routine receives these numbers using a for-range loop. When the channel is closed and all values are received, the loop will terminate automatically. Keep in mind that only a sender can close the channel, to indicate the receiver to not wait for further values from the channel. ## Select Statement for Channels The select statement is used for handling multiple channels. There are a few operations that can be checked with a case statement in the select block. |Case |Channel Operation| |---------|-----------------| | | | |Sending | chan <- value | | | | |Receiving| <- chan | | | | So, we can either check if there is a sender or a receiver available for a channel with a case statement just like a switch statement. ```go package main import ( "fmt" "sync" ) func sendMessage(ch chan string, message string, wg *sync.WaitGroup) { defer wg.Done() ch <- message } func main() { var wg sync.WaitGroup ch1 := make(chan string, 2) ch2 := make(chan string, 2) wg.Add(2) go sendMessage(ch1, "Hello, Gophers!", &wg) go sendMessage(ch2, "Hello, Hamsters!", &wg) go func() { defer wg.Done() wg.Wait() close(ch1) close(ch2) }() ch1 <- "new message to c1" ch2 <- "new message to c2" select { case <-ch1: fmt.Println("Received from ch1") case ch1 <- "new message to c1": fmt.Println("Sent to ch1") case <-ch2: fmt.Println("Received from ch2") case ch2 <- "new message to c2": fmt.Println("Sent to ch2") } } ``` ```bash $ go run channels.go Sent to ch1 $ go run simple.go Received from ch1 $ go run simple.go Received from ch2 $ go run simple.go Sent to ch2 $ go run simple.go Received from ch1 ``` The order of the messages is not guaranteed, the operation which is performed first based on the go routine will be only logged. In the simple example above, we have created two channels `ch1` and `ch2`, and sent two messages to them using two go routines. The main routine then waits for the messages to be received from the channels. We close the channels when the sending is done and simply check for the 4 cases i.e. the send on channel 1, receive on channel 1, and similarly for channel 2. So, that is how we can use the select statement to check which operation is being performed on the different channels, and this forms the basis for the communication between channels. We get more ease in the flow while working with channels. Below is an example to test which `url` or a web server is responding first to the request. ```go package main import ( "fmt" "net/http" "sync" ) func pingGoogle(c chan string, wg *sync.WaitGroup) { defer wg.Done() res, _ := http.Get("http://google.com") c <- res.Status } func pingDuckDuckGo(c chan string, wg *sync.WaitGroup) { defer wg.Done() res, _ := http.Get("https://duckduckgo.com") c <- res.Status } func pingBraveSearch(c chan string, wg *sync.WaitGroup) { defer wg.Done() res, _ := http.Get("https://search.brave.com") c <- res.Status } func main() { gogChan := make(chan string) ddgChan := make(chan string) braveChan := make(chan string) var wg sync.WaitGroup wg.Add(3) go pingDuckDuckGo(ddgChan, &wg) go pingGoogle(gogChan, &wg) go pingBraveSearch(braveChan, &wg) openChannels := 3 go func() { wg.Wait() close(gogChan) close(ddgChan) close(braveChan) }() for openChannels > 0 { select { case msg1, ok := <-gogChan: if !ok { openChannels-- } else { fmt.Println("Google responded:", msg1) } case msg2, ok := <-ddgChan: if !ok { openChannels-- } else { fmt.Println("DuckDuckGo responded:", msg2) } case msg3, ok := <-braveChan: if !ok { openChannels-- } else { fmt.Println("BraveSearch responded:", msg3) } } } } ``` The above example shows how to use a select statement to wait for multiple channels to be ready before proceeding with the next operation. With this example, we can get the channel that sent the response first i.e. which search engine in this case responded to the ping first. Just a bit exaggerated example but it helps in understanding the concept of the `select` statement. ```bash $ go run select-chan.go DuckDuckGo responded: 200 OK Google responded: 200 OK BraveSearch responded: 200 OK $ go run select-chan.go DuckDuckGo responded: 200 OK BraveSearch responded: 200 OK Google responded: 200 OK ``` Let's break each of the steps down: - `pingDuckDuckGo(ddgChan, &wg)` is a method which sends data to the channel `ddgChan`. - `pingGoogle(gogChan, &wg)` is a method that sends data to the channel `gogChan`. - `pingBraveSearch(braveChan, &wg)` is a method that sends data to the channel `braveChan`. - We wait for each go routine to finish using `wg.Wait()` and close the channels. - Finally, we close the channels `gogChan`, `ddgChan`, and `braveChan` to pick up the data from the channel as `<-chan` with the select case block. - The select case will pick the first channel that is ready to receive data. Hence we get the output based on the order of which the channel responded first. - We use the `!ok` condition to check if the channel is closed or not, we have a `openChannels` variable to keep track of the number of open channels, if there are no channels open, we simply break out of the infinite loop. ## Directional Channels Channels can also be designated as "send-only" or "receive-only" to enforce certain communication patterns and enhance safety. This is done by specifying the direction when defining the channel type. ```go package main import ( "fmt" "sync" ) func receiver(ch <-chan int, wg *sync.WaitGroup) { for i := range ch { fmt.Println("Received:", i) } wg.Done() } func sender(ch chan<- int, wg *sync.WaitGroup) { for i := 0; i < 10; i++ { fmt.Println("Sent:", i) ch <- i } close(ch) wg.Done() } func main() { ch := make(chan int) wg := sync.WaitGroup{} wg.Add(2) go receiver(ch, &wg) go sender(ch, &wg) wg.Wait() } ``` In the above example, we have created a channel `ch` and sent 10 values to it using two-go routines. The main routine is waiting for the goroutines to finish before closing the channel. The `sender` sends values `0` through `9`, and the `receiver` prints whenever a value is received. In the `sender` method, we only accept the channel to send data as `chan<-`, and in the `receiver` method, the channel parameter is set to only read from the channel as `<-chan`. ```bash $ go run send-recv.go Sent: 0 Received: 0 Sent: 1 Sent: 2 Received: 1 Received: 2 Sent: 3 Sent: 4 Received: 3 Received: 4 Sent: 5 Sent: 6 Received: 5 Received: 6 Sent: 7 Sent: 8 Received: 7 Received: 8 Sent: 9 Received: 9 ``` When we define a parameter as a write-only channel means that the function can only send data into that channel. It cannot read data from it or close it. This pattern is helpful when you want to make sure that the function is solely responsible for producing data and not consuming or interacting with the channel's current state. When we define a parameter as a read-only channel, it means that the function can only receive data from that channel. It cannot close the channel or send data into it. This pattern is useful when we want to ensure that the function only consumes data from the channel without modifying it or interfering with the sender's logic. Additionally, the compiler will catch code trying to send on a read-only channel or receive from a write-only one. ## Common Channel Usage Pattern There are a variety of ways in which channels can be used in Go. In this section, we'll explore some of the most common patterns for using channels in Go. Some of the most useful and idiomatic channel usage patterns include pipelines, fan-in and fan-out, etc. ### Async Await pattern for Channels In Go, goroutines and channels enable an elegant async/await style. A goroutine can execute a task asynchronously, while the main thread awaits the result using a channel. The async-await pattern in Go involves initiating multiple tasks concurrently, each with its own goroutine, and then awaiting their completion before proceeding. Channels are used to communicate between these goroutines, allowing them to work independently and provide results to the main routine when ready. ```go package main import ( "fmt" "net/http" ) func fetchURL(url string, ch chan<- http.Response) { go func() { res, err := http.Get(url) if err != nil { panic(err) } defer res.Body.Close() ch <- *res }() } func task(name string) { fmt.Println("Task", name) } func main() { fmt.Println("Start") url := "http://google.com" respCh := make(chan http.Response) fetchURL(url, respCh) task("A") task("B") response := <-respCh fmt.Println("Response Status:", response.Status) fmt.Println("Done") } ``` ```bash $ go run async.go Start Task A Task B Response Status: 200 OK Done ``` In the above example, we have created a function `fetchURL` which takes a URL and a channel as an argument. The channel `respCh` is used to communicate between the goroutines. The function fires up a goroutine that fetches the request, we send a `GET` request to the provided URL and send the response to the provided channel. In the main function, we access the `response` by receiving the data from the channel as `<-respCh`. Before doing this, we can do any other task simultaneously, like `task("A")` and `task("B")` which just prints some string(it could be anything). But this should be before we pull in from the channel, anything after the access will be blocked i.e. will be executed sequentially. ### Pipeline pattern for Channels The pipeline pattern is used to chain together a sequence of processing stages, each stage consumes input, processes data, and passes the output to the next stage. This type of pattern can be achieved by chaining different channels from one go routine to another.  So, the pipeline pattern using channels in Go, data flows sequentially through multiple stages: Stage 1 reads input and sends to Channel A, Stage 2 receives from Channel A and sends to Channel B, and Stage 3 receives from Channel B and produces the final output. ```go package main import ( "fmt" "sync" ) func generate(nums []int, out chan<- int, wg *sync.WaitGroup) { fmt.Println("Stage 1") for _, n := range nums { fmt.Println("Number:", n) out <- n } close(out) wg.Done() } func square(in <-chan int, out chan<- int, wg *sync.WaitGroup) { fmt.Println("Stage 2") for n := range in { sq := n * n fmt.Println("Square:", sq) out <- sq } close(out) wg.Done() } func print(in <-chan int, wg *sync.WaitGroup) { for n := range in { fmt.Println(n) } wg.Done() } func main() { input := []int{1, 2, 3, 4, 5} var wg sync.WaitGroup wg.Add(3) stage1 := make(chan int) stage2 := make(chan int) go generate(input, stage1, &wg) go square(stage1, stage2, &wg) go print(stage2, &wg) wg.Wait() } ``` In the above example, we have created a sequence of processing stages, each stage consumes input, processes data, and passes the output to the next stage. We can consider the functions `generate`, `square`, and `print` as stages `1`, `2`, and `3` respectively. - The generate function, takes in the input as a slice of integers, an unbuffered channel, and the waitgroup b reference, the function basically iterates over the numbers in the slice and sends it to the channel provided in the parameters. - The square function takes in the stage1 channel that the channel from the stage1, as well as its own channel as `stage2` (remember the `stage1` channel has sent the numbers via the generating function). - The square function then iterates over the number sent from the channel `stage1` as `in` and squares it and sends it to the channel provided as `stage2` as the `out` channel. - The print function takes in the stage2 channel as an argument and iterates over the number sent from the channel `stage2` and prints it. ``` $ go run pipeline.go Stage 1 Number: 1 Stage 2 Square: 1 1 Number: 2 Number: 3 Square: 4 Square: 9 Number: 4 4 9 Square: 16 16 Number: 5 Square: 25 25 ``` So, we can see the order of the execution, both the pipelines started synchronously, However, they perform the operation only when the data is sent from the previous channel. We first print the `number` from the `generate` function, then print the squared value in the `square` function, and finally print it as `Square: value` in the print function. ### Fan-In pattern for Channels The Fan-In pattern is used for combining data from multiple sources into a single stream for unified processing, often using a shared data structure to aggregate the data. We can create the fan-in pattern by merging multiple input channels into a single output channel.  The fan-in pattern is when multiple input channels (A, B, C) are read concurrently, and their data is merged into a single output channel (M). ```go package main import ( "fmt" "io/ioutil" "sync" ) func readFile(file string, ch chan<- string) { content, _ := ioutil.ReadFile(file) fmt.Println("Reading from", file, "as :: ", string(content)) ch <- string(content) close(ch) } func merge(chs ...<-chan string) string { var wg sync.WaitGroup out := "" for _, ch := range chs { wg.Add(1) go func(c <-chan string) { for s := range c { out += s } wg.Done() }(ch) } wg.Wait() return out } func main() { ch1 := make(chan string) ch2 := make(chan string) go readFile("data/f1.txt", ch1) go readFile("data/f2.txt", ch2) merged := merge(ch1, ch2) fmt.Println(merged) } ``` In the above example, the `readFile` function reads the contents of the file and sends it to the channels `ch1` and `ch2` from different go routines. The `readFile` takes in the channel as send only channel which reads the file and sends the content to the channel as `ch <- string(content)`. The `merge` function takes in `2` it can also be `n` number of channels to parse from as indicated as `...<-chan`, it iterates over each channel, and for each channel, it reads the contents, and appends as a single string. ```bash $ go run fan-in.go Reading from data/f1.txt as :: This is from file 1 Reading from data/f2.txt as :: This is from file 2 This is from file 1 This is from file 2 $ go run fan-in.go Reading from data/f2.txt as :: This is from file 2 Reading from data/f1.txt as :: This is from file 1 This is from file 2 This is from file 1 ``` So, this is how the fan-in pattern works, We create multiple channels and combine the results into a single stream of data(in this example a single string). ### Fan-Out Pattern for Channels The Fan-Out pattern involves taking data from a single source and distributing it to multiple workers or processing units for parallel or concurrent handling. Fan-out design splits an input channel into multiple output channels, it is used to distribute branches of work or data across concurrent processes.  The fan-out pattern is when data from a single input channel (A) is distributed to multiple worker channels (X, Y, Z) for parallel processing. ```go package main import ( "fmt" "os" "sync" ) func readFile(file string, ch chan<- string, wg *sync.WaitGroup) { defer wg.Done() content, err := os.ReadFile(file) if err != nil { fmt.Printf("Error reading from %s: %v ", file, err) return } ch <- string(content) } func main() { files := []string{"data/f1.txt", "data/f2.txt"} var wg sync.WaitGroup ch := make(chan string) for _, f := range files { wg.Add(1) go readFile(f, ch, &wg) } go func() { wg.Wait() close(ch) }() var fileData []string for content := range ch { fileData = append(fileData, content) } fmt.Printf("Read %d files ", len(fileData)) fmt.Printf("Contents: %s", fileData) } ``` In the above example, we create a single channel `ch` as our single source, we loop over all the files, and create go routines calling the `readFile` function. The `readFile` function takes in the filename, channel, and the WaitGroup reference, the function reads the file and sends the content to the channel as `ch <- content`. The `readFile` is called concurrently for all the files, Here we have done a fan-out of the task into multiple go routines, then in the main function, we iterate over the channel and receive the content. ```bash $ go run fan-out.go Read 2 files Contents: [This is from file 2 This is from file 1 ] $ go run fan-out.go Read 2 files Contents: [This is from file 1 This is from file 2 ] ``` Here's a brief summary of the fan-out pattern from the example provided: - Multiple files are read concurrently using goroutines. This "fans out" the work. - The `readFile` function runs in a goroutine to process each file separately. - WaitGroup coordinates the goroutines. - A shared channel ch collects the results from each goroutine. - The main goroutine reads from the channel and aggregates the results. - Channel is closed and ranged over to collect results cleanly. I have a few more patterns to demonstrate that have been provided in the GitHub on the [100 days of Golang](https://github.com/Mr-Destructive/100-days-of-golang/tree/main/scripts/channels/patterns/) repository. That's it from the 31st part of the series, all the source code for the examples are linked in the GitHub on the [100 days of Golang](https://github.com/Mr-Destructive/100-days-of-golang/tree/main/scripts/channels) repository. ## References - [Channels](https://go.dev/ref/spec#Channel_types) - [Effective Go: Channels](https://go.dev/doc/effective_go#channels) - [Fan-In and Fan-Out](https://mariocarrion.com/2021/08/19/learning-golang-concurrency-patterns-fan-in-fan-out.html) - [Go 101: Channels](https://go101.org/article/channel.html) ## Conclusion So, from this part of the series, we were able to understand the fundamentals of channels in golang. By using the core concepts from the previous posts like go routines and wait groups, we were able to work with channels in golang. We wrote a few examples for different patterns using concurrency concepts with channels. Patterns like pipelines, fan-in, fan-out, async, and some usage of select statements for channels were explored in this section. Hopefully, you found this section helpful, if you have any comments or feedback, please let me know in the comments section or on my social handles. Thank you for reading. Happy Coding :) https://www.meetgor.com/posts/golang-go-routines Introduction One of the key features that set Go apart from many other languages is its native support for Goroutines - lightweight concurrent functions that ca Sat, 29 Jul 2023 00:00:00 UTC ## Introduction One of the key features that set Go apart from many other languages is its native support for Goroutines - lightweight concurrent functions that can run concurrently and efficiently manage concurrency tasks. Goroutines are an essential aspect of Go's concurrency model, enabling developers to build highly concurrent and performant applications effortlessly. In this blog post, we will dive into the world of Goroutines and explore how they work, why they are essential for concurrent programming in Go, and how they can greatly improve the responsiveness and efficiency of your applications. This post will cover go routines and a primer on wait groups, in the next article we will be looking deeply into channels where all these three things can be demonstrated and understood in a better and more useful way. ## Concurrency and Parallelism This two concepts are quite crucial before diving into the fundamentals of concurreny. **Concurrency** is about dealing with multiple things at once. Concurrent programs can have several processes or threads running simultaneously on a single CPU core by rapidly switching between them (context switching). The threads are interleaved, not necessarily executing at literally the same time. The CPU can switch between these tasks to give the appearance of simultaneous progress. **Parallelism** is about doing multiple things at literally the same time. Parallel programs can execute multiple computations simultaneously on separate CPU cores. The threads actually execute in parallel. ## What is a Go Routine A go routine is a simple lightweight thread managed by the Go runtime. In the simplest of terms, a go routine can be defined as: > Go routine is a way to perform multiple tasks within a program, allowing different parts of the program to work simultaneously and make the most out of the resources. Also it can be stated as: A goroutine in Golang is a lightweight, independently executing function that runs concurrently with other goroutines within the same address space. In other words, it is a concurrent unit of execution. Focussing on the word **same address space** that will be really critical in the later sections of this article. ## Features of Go Routines Go routines form a key part of Go's concurrency model. Here are some of the key features of go routines: - **Lightweight Thread**: A Go routine is often referred to as a lightweight thread. - **Independent Execution**: Go routines run independently of each other, enabling concurrent execution. - **Managed by Go**: Go routines are managed by the Go runtime, making them easy to use. - **Low Overhead**: Go routines have low memory overhead, allowing us to create thousands of them efficiently. - **Communication**: Go routines can communicate and synchronize data through channels. - **Asynchronous**: Go routines can execute asynchronously, allowing other parts of the program to continue running. - **Scalability**: Go routines are the foundation of scalable concurrent programming in Go. Unlike threads in other languages, goroutines are cheap and you can easily create thousands or even millions of them in a program. ## Example of Go Routines Creating a go routine is not hard, just add the keyword `go` before the function call and the go runtime will create a new go routine inside the main function or wherever is the context from. Remember, the main function is also a go routine. ```go package main import ( "fmt" "time" ) func process() { fmt.Println("Hello World!") time.Sleep(time.Second) } func main() { start := time.Now() go process() go process() end := time.Now() duration := end.Sub(start) fmt.Println(duration) } ``` ```bash scripts/go-routines on  main via 🐹 v1.20 $ go run main.go Hello World! 15.607µs scripts/go-routines on  main via 🐹 v1.20 $ go run main.go 9.889µs scripts/go-routines on  main via 🐹 v1.20 $ go run main.go 8.834µs scripts/go-routines on  main via 🐹 v1.20 $ go run main.go 9.158µs scripts/go-routines on  main via 🐹 v1.20 $ go run main.go 12.54µs scripts/go-routines on  main via 🐹 v1.20 $ go run main.go Hello World! Hello World! 10.19µs scripts/go-routines on  main via 🐹 v1.20 $ go run main.go 14.1µs scripts/go-routines on  main via 🐹 v1.20 $ ``` Quite unpredictable output right? This is a power of go-routines. It is **asynchronous** so it will not block the main function. The two function calls to `process`, are executed independently of the main function scope. The program just calls the process function and captures the output, and sequentially reaches the end of the program(main function), At this point the go routines(threads) inside the main function are stopped abruptly. Let's break it down if it is not clear yet. - The main function starts. - `go process()` creates a Go routine and starts its execution. - Meanwhile, another call to `go process()` creates a separate Go routine and starts its execution as well. - Meanwhile, calculate the time difference between the main function start and end. - Meanwhile main function ends. So, in summary, the main function is only able to capture the output of the `duration` as it is synchronous and if the process has been executed, it prints the `Hello World!` message. Hence we have different outputs because the uncontrolled concurrency, lack of coordination, and OS scheduling across the program runs are different. ## Wait Groups In simple terms, WaitGroup is used to synchronize multiple goroutines and to wait for them to finish executing. This allows the go routines to be completed before the completion of the main function, hence it blocks the main function from leaving/exiting the scope. > A WaitGroup is a synchronization primitive that allows a goroutine to wait for a collection of other goroutines to finish executing. - A WaitGroup is initialized with a counter representing the number of goroutines to wait for. - The Add() method increments the counter by the given value. This is called by each goroutine to indicate it is running. - The main goroutine calls Add() to set the initial count, then launches worker goroutines. - A WaitGroup is typically passed by a pointer to goroutines that need to be waited on. - The Done() method decrements the counter by 1. Goroutines call this when finished. - Each worker calls Done() when finished, decrementing the counter. - The Wait() method blocks until the counter reaches 0, indicating all goroutines have finished. - Main calls Wait() to block until Done() brings counter to 0. This provides a simple way to synchronize multiple goroutines finishing their work with a main thread that needs to wait for them to complete. The counter ensures the main thread knows how many goroutines it is waiting for. Interacting and working with go routines with synchronization using a wait group is quite intuitive and simple to follow. Let's look at a simple example below: ```go package main import ( "fmt" "sync" "time" ) func process(pid int, wg *sync.WaitGroup) { fmt.Printf("Started process %d ", pid) time.Sleep(1 * time.Second) fmt.Printf("Completed process %d ", pid) defer wg.Done() } func main() { now := time.Now() var wg sync.WaitGroup for i := 0; i < 10; i++ { wg.Add(1) go process(i, &wg) } wg.Wait() fmt.Println("All processes completed") end := time.Now() fmt.Println(end.Sub(now)) } ``` ```bash scripts/go-routines on  main [?] via 🐹 v1.20 $ go run wg.go Started process 9 Started process 0 Started process 1 Started process 2 Started process 3 Started process 4 Started process 5 Started process 6 Started process 7 Started process 8 Completed process 8 Completed process 3 Completed process 9 Completed process 0 Completed process 1 Completed process 2 Completed process 5 Completed process 4 Completed process 6 Completed process 7 1.000563257s All processes completed scripts/go-routines on  main [?] via 🐹 v1.20 $ ``` In the above example, we have used the same function `process` but with a slight twist, we have added a process id, just an integer to represent the go routine. We print the start and completion of the function in between the sleep for 1 second. We also have a wait group. - A [WaitGroup](https://pkg.go.dev/sync#WaitGroup) is basically a struct type defined in the [sync](https://pkg.go.dev/sync) package. - The variable `wg` is a new wait group instance that will be used to synchronize and wait for the completion of groups of go routines. We create a for loop with `10` iterations, so as to spawn 10 `process` function calls. We first use the `wg.Add(1)` which says to the wait group to wait for 1 go routine. The immediate next line is a go routine `go process()` which takes in a `pid` just to track which go routine is being executed in the loop. Inside the `process` function, we simply say the process with the given `pid` has started as a print statement, sleeps for a second, and then print the end of the process with `pid`. The end of the function is marked with `defer wg.Done()`, this is to indicate that the go routine is completed. The wg(wait group) has a counter that keeps track of the number of go routines it has to synchronize or wait for them to complete. In the [Add](https://pkg.go.dev/sync#WaitGroup.Add) function, the internal counter in the `WaitGroup` is incremented by the `delta` the integer parsed as a parameter. And the [Done](https://pkg.go.dev/sync#WaitGroup.Done) function, decrements the internal counter in the `WaitGroup` which indicates that the go routine is completed. In the main function, we call the [wg.Wait](https://pkg.go.dev/sync#WaitGroup.Wait) which will block until the counter for the `WaitGroup` is 0 i.e. all go routines have completed their execution. So, we created 10 go routines which ran concurrently but were synchronized with the help of WaitGroups. The WaitGroup is allowed to block the main function till all the go routines are done executing. ## Go Routine with WaitGroup Use Cases Go routines can be used for creating asynchronous tasks, and also for creating concurrent tasks. By using wait groups, we can create a way to wait for multiple goroutines to complete. By using go routines and wait groups, we can complete n number of tasks in a time frame of 1 task's completion. However, to create concurrent communication between the other processes, we need `channels` (which we will explore in the next article). Here is a simple breakdown of what asynchronous and concurrent tasks might refer to: **Asynchronous** tasks run independently of the main program flow, allowing the main program to continue executing without waiting for the task's completion. For example, not blocking other tasks in the sequential flow of the main function. **Concurrent** tasks run simultaneously and can execute at the same time as other tasks. They make use of multiple threads (goroutines in Go's case) to achieve parallel execution. For example, running multiple tasks parallel will cut the time down for spinning each task after the completion of another task. Some of the asynchronous tasks that can be done with go routines might include the following: - Sending a mail while saving the user to the database. - Fetch and process data from multiple websites(web scraping/crawling). - High-performance message brokers and queuing systems for inter-process communication. One practical way of using go routines with wait groups would be to send mail, we are not going to see the actual implementation of the mail-sending stuff. However, we can try to mimic how the setup might be for sending bulk mail. By creating a wait group and a list of mail addresses to send, a go routine can be created with a function that sends those emails. ```go package main import ( "fmt" "sync" ) func sendMail(address string, wg *sync.WaitGroup) { fmt.Println("Sending mail to", address) defer wg.Done() // Actual mail sending, smtp stuff // handle errors // client, err := smtp.Dial("smtp.example.com:587") // errr = client.Mail("sender@example.com") // err = client.Rcpt(address) // wc, err := client.Data() //_, err = wc.Write([]byte("This is the email body.")) // err = wc.Close() // client.Quit() } func main() { emails := []string{ "recipient1@example.com", "recipient2@example.com", "xyz@example.com", } wg := sync.WaitGroup{} wg.Add(len(emails)) for _, email := range emails { go sendMail(email, &wg) } wg.Wait() fmt.Println("All emails queued for sending") // Do other stuff } ``` In the above example, the `emails` is a list of email ids to send the mail. We have created a wait group that we can initialize with the total number of go routines probably to be executed. The `wg.Add` method is parsed with the number of emails to be sent hence that equals the go routines to spawn. So, we can then in the for loop, iterate over each mail and send the emails with the `sendMail` function as go routine. The `wg.Wait` function outside the loop, will make sure the main function is halted before all the go routines complete their execution. There is one more way to call a function as a go routine without changing its signature since we had to pass `wg` pointer as the WaitGroup reference to it to acknowledge the completion of the go routine. We could wrap these two operations wiz. calling the function and calling the `wg.Done` method with an anonymous function. ```go package main import ( "fmt" "sync" ) func sendMail(address string) { fmt.Println("Sending mail to", address) } func main() { emails := []string{ "recipient1@example.com", "recipient2@example.com", "xyz@example.com", } wg := sync.WaitGroup{} wg.Add(len(emails)) for _, email := range emails { mail := email go func(m string) { sendMail(m) wg.Done() }(mail) } wg.Wait() fmt.Println("All emails queued for sending") // Do other stuff } ``` This does the exact same thing, but we don't have to change the signature of the function, This keeps the functional logic of the function in its place and handles the concurrency on the go. ```go for _, email := range emails { mail := email go func(m string) { sendMail(m) wg.Done() }(mail) } ``` If the above bit scares you, don't worry it's too simple. - We are iterating over the email slice using a for loop and creating a Go routine for each email address. The loop variable email represents the email address at the current iteration. - However, to avoid the loop variable capture issue (where all Go routines would share the same email variable), we create a new variable mail and assign the value of email to it. This step ensures that each Go routine captures its own copy of the email address. - We immediately create an anonymous function (a closure) using the go keyword. This anonymous function takes the `mail` variable as a parameter m and is executed concurrently as a Go routine. Inside the Go routine, we call the `sendMail` function with the email address `m`. - After the `sendMail` call has been executed i.e. email has been sent, we call wg.Done() to notify the WaitGroup that the Go routine has completed its work. This allows the WaitGroup to properly synchronize and wait for all Go routines to finish before the program proceeds beyond wg.Wait() in the main function. This is one way to do it if you want to separate the mail-sending logic from the goroutines/concurrency task. However, this should be handled with care as the variables inside the closure might be shared among all the goroutines instead of having individual variable literals. To ensure that each goroutine operates on its own copy of the email address, we use the approach of creating a new variable mail and passing it as a parameter to the anonymous function. This way, each goroutine captures its unique email address, avoiding any interference or unintended sharing of data between goroutines. ## Mutual Exclusion Locks In the previous examples, we saw how goroutines and wait groups allow us to run multiple tasks concurrently in Go. However, sometimes these concurrent goroutines need to access shared resources like memory, files, network sockets, etc. When more than one goroutine tries to access a resource at the same time, it can lead to **race conditions** and unpredictable behavior. To handle this, we need a way to ensure only one goroutine can access the resource at a time. This is where mutual exclusion locks come in. A **mutual exclusion lock**, or **mutex**, provides a mechanism to lock access to a shared resource. It ensures only one goroutine at a time can acquire the lock, blocking other goroutines until the lock is released. For example, say we have multiple goroutines trying to append data to the same memory buffer(could be file/database/etc.) concurrently: ```go package main import ( "fmt" "os" "sync" ) func WriteToFile(filename string, contents string, buffer *[]byte, wg *sync.WaitGroup) { defer wg.Done() *buffer = append(*buffer, []byte(contents)...) err := os.WriteFile(filename, *buffer, 0644) if err != nil { fmt.Println(err) } } func main() { var wg sync.WaitGroup var sharedBuffer []byte wg.Add(2) go WriteToFile("data/f1.txt", "Hello ", &sharedBuffer, &wg) go WriteToFile("data/f1.txt", "World! ", &sharedBuffer, &wg) wg.Wait() fmt.Println(string(sharedBuffer)) } ``` ```bash $ go run --race no-mutex.go ================== WARNING: DATA RACE Read at 0x00c000012030 by goroutine 8: main.WriteToFile() /home/meet/code/100-days-of-golang/scripts/go-routines/no-mutex.go:11 +0xe7 main.main.func2() /home/meet/code/100-days-of-golang/scripts/go-routines/no-mutex.go:24 +0x64 Previous write at 0x00c000012030 by goroutine 7: main.WriteToFile() /home/meet/code/100-days-of-golang/scripts/go-routines/no-mutex.go:11 +0x16a main.main.func1() /home/meet/code/100-days-of-golang/scripts/go-routines/no-mutex.go:23 +0x64 Goroutine 8 (running) created at: main.main() /home/meet/code/100-days-of-golang/scripts/go-routines/no-mutex.go:24 +0x1f6 Goroutine 7 (finished) created at: main.main() /home/meet/code/100-days-of-golang/scripts/go-routines/no-mutex.go:23 +0x14e ================== ``` This is because we are trying to access the same memory address at the same time. This is a race condition and can lead to unpredictable behavior. Try removing the `--race` flag while running, in this little stupid example, it might not be obvious, but in complex and constrained environments, this can get the application in serious trouble. **NOTE**: We are using `go run --race no-mutex.go` to check if there are any race conditions in the program. This is the [race detector](https://go.dev/doc/articles/race_detector) flag in the run command. To avoid this race condition, we need to add the mutex locks provided in the [sync.Mutex](https://pkg.go.dev/sync#Mutex) type. There are methods like [Lock](), [Unlock](), and [TryLock]() which help in locking access of the resource to a single entity at a given time. When a goroutine calls `Lock()` on a mutex, it acquires the lock. If the mutex is already locked by another goroutine, the calling goroutine will be blocked (put to sleep) until the lock becomes available. Once the lock is acquired successfully, the goroutine can proceed with its critical section, which is the part of the code that should not be executed concurrently by multiple goroutines. When a goroutine calls `Unlock()` on a mutex, it releases the lock. This allows other waiting goroutines to acquire the lock and execute their critical sections. It's essential to ensure that `Unlock()` is called after the critical section to release the mutex and avoid deadlocks. The critical section/shared resource should not be accessed after the release of this lock. ```go package main import ( "fmt" "os" "sync" ) func WriteToFile(filename string, contents string, buffer *[]byte, wg *sync.WaitGroup, mutex *sync.Mutex) { defer wg.Done() contentBytes := []byte(contents) mutex.Lock() *buffer = append(*buffer, contentBytes...) f, err := os.OpenFile(filename, os.O_APPEND|os.O_WRONLY, 0644) if err != nil { fmt.Println(err) } defer f.Close() _, err = f.Write(contentBytes) if err != nil { fmt.Println(err) } mutex.Unlock() } func main() { var wg sync.WaitGroup var mut sync.Mutex var sharedBuffer []byte wg.Add(2) go WriteToFile("data/f1.txt", "Hello Gophers! ", &sharedBuffer, &wg, &mut) go WriteToFile("data/f1.txt", "Welcome to Goworld! ", &sharedBuffer, &wg, &mut) wg.Wait() fmt.Println(string(sharedBuffer)) } ``` ```bash $ go run --race mutex.go Welcome to Goworld! Hello Gophers! $ go run --race mutex.go Hello Gophers! Welcome to Goworld! $ go run --race mutex.go Hello Gophers! Welcome to Goworld! $ go run --race mutex.go Welcome to Goworld! Hello Gophers! ``` The above example is a preventive measure for race conditions with a mutex Lock on the shared resource used in the go routines. Let's break down the code step by step. First, we initialize a few variables: - `wg` as `sync.WaitGroup`: The `wg` is a waitgroup that will be used for the synchronization of the go-routines by blocking the main function to exit - `mut` as `sync.Mutex`: The `mut` is a structure that internally holds a few integer values indicating either a blocked or unblocked state. The `sync.Mutex` has two private fields wiz. `state` and `sema`, the state holds the mutex state either `0`(unlocked) or `1` as locked. The `sema` field is a `uint32` that is used for blocking and signaling, it acts as a semaphore to manage blocking and unblocking goroutines trying to acquire the mutex. This will be used to acquire `Lock` and `Unlock` on the shared resource while writing the data to the file or appending the data to the resource. - `sharedBuffer` as `[]byte`: The `sharedBuffer` is the actually shared resource that will be used to hold the strings for keeping track of the data written to the file. It will be the resource that would require to lock for mutating its value(appending to the slice) among the go routines. We add `2` to the `wg` to indicate to wait for the two go-routines to complete, in the next line we call two go routines as the function `WriteToFile`. The `WriteToFile` is a function that takes in quite a few parameters namely, the filename, the content to write, the reference to the sharedBuffer, waitgroup, and the mutex. Inside the function `WriteToFile`: - We first `defer` the `waitgroups` as `Done` i.e. to call the `wg.Done` method at the end of the function call. - Typecast the `contents` from `string` as the `[]byte`. - Acquire the `mutex.Lock()` i.e. to say "The below operations should not be done concurrently, one at a time". We then append the `contents` to the `buffer` which is a pointer to the `sharedBuffer` in the main function, so essentially we are trying to mutate the `sharedBuffer` in this function. - Open the file as `O_APPEND` and `O_WRONLY` to indicate that the file should be opened in append/writing mode. (We have observed this type of operation in the [Golang: File Write](https://www.meetgor.com/golang-file-write/) article) - Use the `Write` method to write the slice of bytes(contents) into the file that we opened. We have a `defer` for the closing of the file. We obviously check for errors and print them, but it could be a `panic` or `log` depending on the type of operation the application is doing. So that is the all operation we want to do, so we finally open the lock with `mutex.Unlock()` which will allow the other go routine if any to access the resource and proceed with the operations. ### Read Write Mutual Exclusion Lock The mutual exclusion lock is good if you have a write operation-heavy application. However, if we have read write operations in more or the less in same proportion(read heavy) we don't want to have readers getting blocked when other readers are accessing the resource since it is not a mutation process. We could allow many readers to read simultaneously. But for the writing operation, we want to block the readers/writers. The writer should be given preference first in case of a lock by other writers. This would prevent a writer from waiting for readers to complete. This is usually referred to as **Read Write Mutual Exclusion**. ```go package main import ( "fmt" "sync" "time" ) func reader(id int, count *int, wg *sync.WaitGroup, mutex *sync.Mutex) { defer wg.Done() if !mutex.TryLock() { fmt.Printf("Reader %d blocked! ", id) return } defer mutex.Unlock() fmt.Printf("Reader %d: read count %d ", id, *count) } func writer(id, increment int, count *int, wg *sync.WaitGroup, mutex *sync.Mutex) { defer wg.Done() defer mutex.Unlock() mutex.Lock() *count += increment time.Sleep(5 * time.Millisecond) fmt.Printf("Writer %d: wrote count %d ", id, *count) } func main() { count := 1 var wg sync.WaitGroup var mutex sync.Mutex readers := 5 writers := 3 wg.Add(readers) for i := 0; i < readers; i++ { go reader(i, &count, &wg, &mutex) } wg.Add(writers) for i := 0; i < writers; i++ { go writer(i, 1, &count, &wg, &mutex) } wg.Wait() } ``` ```bash $ go run --race no-rwmutex.go Reader 0: read count 1 Reader 3 blocked! Reader 2 blocked! Reader 1 blocked! Reader 4 blocked! Writer 0: wrote count 2 Writer 1: wrote count 3 Writer 2: wrote count 4 $ go run --race no-rwmutex.go Reader 2 blocked! Reader 0: read count 1 Reader 1 blocked! Reader 4 blocked! Reader 3 blocked! Writer 2: wrote count 2 Writer 0: wrote count 3 Writer 1: wrote count 4 ``` The above example has a `reader` and a `writer` method, the `reader` method simply has to read the shared resource `count`. It acquires a `Mutex` lock before reading and unlocks it thereafter. Similarly, the `writer` function is used for incrementing the `count` shared resource. The `reader` method has a [TryLock](https://pkg.go.dev/sync#Mutex.TryLock) method that tries to acquire a mutex lock on the resource, if the resource is already locked, the function will return `false` and hence we can say that the reading is blocked(just for demonstration). And if the function `TryLock` returns `true`, it will acquire the `Lock`. We further `defer` the `Unlock` and access the `count` variable which is passed as a reference. The `writer` method is simply acquiring the `Lock` and incrementing the `counter` and thereafter `Unlock` is called with `defer`. In the above code: - The `reader` and the `writer` both might be waiting for the lock to be released, however, for readers to wait for reading doesn't make sense. - Because if you would want to just read a particular memory address, there shouldn't be any locks for one reader to wait for other readers to finish. - However, for writing, there has to be a lock. The mutex lock will lock the resource irrespective of the `reader` or `writer`. This might not be as visible here, but it might be the reason, that all the `readers` are blocked from reading due to another reader's or writer's lock. The `sync` package has the [RWMutex](https://pkg.go.dev/sync#RWMutex) that does this exact same thing. It is almost similar to the `Mutex` however, it would allow concurrent reading operation and prefer writing operation before readers to prevent writer starvation. ```go package main import ( "fmt" "sync" "time" ) func reader(id int, count *int, wg *sync.WaitGroup, mutex *sync.RWMutex) { defer wg.Done() defer mutex.RUnlock() if !mutex.TryRLock() { fmt.Printf("Reader %d blocked! ", id) return } fmt.Printf("Reader %d: read count %d ", id, *count) } func writer(id, increment int, count *int, wg *sync.WaitGroup, mutex *sync.RWMutex) { defer wg.Done() defer mutex.Unlock() mutex.Lock() *count += increment time.Sleep(5 * time.Millisecond) fmt.Printf("Writer %d: wrote count %d ", id, *count) } func main() { count := 1 var wg sync.WaitGroup var mutex sync.RWMutex readers := 5 writers := 3 wg.Add(readers) for i := 0; i < readers; i++ { go reader(i, &count, &wg, &mutex) } wg.Add(writers) for i := 0; i < writers; i++ { go writer(i, 1, &count, &wg, &mutex) } wg.Wait() } ``` ```bash $ go run --race rwmutex.go Reader 0: read count 1 Reader 3: read count 1 Reader 1: read count 1 Reader 2: read count 1 Reader 4: read count 1 Writer 1: wrote count 2 Writer 0: wrote count 3 Writer 2: wrote count 4 Writer 3: wrote count 5 Writer 4: wrote count 6 $ go run --race rwmutex.go Reader 1 blocked! Reader 0: read count 1 Reader 2 blocked! Reader 3 blocked! Reader 4 blocked! Writer 4: wrote count 2 Writer 0: wrote count 3 Writer 1: wrote count 4 Writer 3: wrote count 5 Writer 2: wrote count 6 ``` In the modified example, all the logic remains the same, just the `sync.Mutex` is replaced with `sync.RWMutex`. Also for trying to acquire the lock in the `reader` method [TryRLock](https://pkg.go.dev/sync#RWMutex.RLock) is used instead of [TryLock](https://pkg.go.dev/sync#RWMutex.TryRLock) which will check if the existing lock acquired is of a reader or writer, if it is a reader, it will return `true`, else `false`. Also the `Unlock` is replaced with the `RUnlock` method for releasing the read lock. In the `writer` method, everything remains the same so the writer has to acquire the lock irrespective of whether the current lock is from the reader/writer, so it is a normal `Lock` and `Unlock`. In the above example, we can see all the read operations sometimes are executed instead of getting blocked. This is due to the `RWMutex` Lock and Unlock on the read operation/function. - When one reader is reading, it can't block other readers. - However with simple `Mutex`, the reader is even blocked when another reader is reading. - For the write operation, it will be blocked as usual, so if a writer is performing a write operation and a reader/readers come in, they will be blocked, also if in the meantime, while the resource is locked, another writer comes in, the writer will be given preference instead of waiting for all readers to complete. This prevents writer starvation. You can see, that the readers are still blocked the majority of the time, however, they are blocked due to the writer locking the resource and not any other reader. There is a bit more suitable [example](https://github.com/Mr-Destructive/100-days-of-golang/blob/main/scripts/go-routines/file-rw.go) related to `file` reading and writing with `RWMutex`, make sure to check that out to get a more clear understanding of an actual case of using the `RWMutex`. ### Channels This is a big part and I would like to delve into this topic in a separate post. There are some patterns like `fan-in`, `fan-out`, `worker-pool`, `pub-sub`, etc. which are really common in web applications and backend systems. These patterns we shall explore in the next article. Channels are a way to provide a safe and idiomatic way for Goroutines to exchange data and coordinate their execution without resorting to low-level mechanisms like shared memory or explicit locking. That's it from the 30th part of the series, all the source code for the examples are linked in the GitHub on the [100 days of Golang](https://github.com/Mr-Destructive/100-days-of-golang/tree/main/scripts/go-routines) repository. ### References: - [How Goroutines Work](https://blog.nindalf.com/posts/how-goroutines-work/) - [Concurrency patterns in Golang: WaitGroups and Goroutines](https://blog.logrocket.com/concurrency-patterns-golang-waitgroups-goroutines/) - [A complete journey with Goroutines](https://riteeksrivastava.medium.com/a-complete-journey-with-goroutines-8472630c7f5c) ## Conclusion From this part of the series, the fundamentals of golang's concurrency model were understood specifically spawning go-routines, synchronously executing go-routines with the help of a wait group, mutex locks, and how to secure concurrent access to a shared resource. In the next part of the series, these concepts will be used in asynchronous communication using channels. Hopefully, you have got the basics of concurrency in golang cleared from this post. If you have any queries, suggestions, or feedback, please feel free to comment below or contact me on the social handles. Thank you for reading. Happy Coding :) https://www.meetgor.com/posts/golang-generics Introduction In the 29th post of the series, we will be looking into generics in Golang. Generics were added in Golang version 1.18, so they are quite new in th Sun, 23 Jul 2023 00:00:00 UTC ## Introduction In the 29th post of the series, we will be looking into generics in Golang. Generics were added in Golang version 1.18, so they are quite new in the world of Golang but the concept is quite old in other programming languages. Generics provide a powerful toolset for writing more expressive and concise code that can handle a wide range of data types. With generics, we can write reusable algorithms, data structures, and functions that work seamlessly with various types, without sacrificing type safety. We will learn how to create generic functions and work with generic types. Additionally, we will cover type constraints and interfaces, which allow us to specify requirements for the types used with our generics. ## Generic Type in Functions We can define a generic type with the keyword `any` that is going to replace the type `T` i.e. any data type with the inferred type at compilation. This makes the code reusable to any relevant data type to be used for that operation/task. We can provide the type `any` after the name of the function/struct to make it generic like `func Name[T any](x T)`. Here, the Name is a function that takes in a generic type `T` it could be any type and the variable is named as `x` that could be used inside the function. We could also make the function take specific types instead of `any` but we will eventually move into that. However, for now, let's ease the process of learning and then move on to the optimizations and adding constraints. ```go package main import ( "fmt" ) func Print[T any](stuff T) { fmt.Println(stuff) } func main() { Print("hello") Print(123) Print(3.148) } ``` [GO Playground Link](https://go.dev/play/p/t-ODmkHu5BJ) ```bash $ go run main.go hello 123 3.148 ``` The above is the simplest example that could be used to demonstrate a generic function. The function `Print` takes a parameter `stuff` of a generic type denoted by a type parameter `T`. The type parameter `T` serves as a placeholder that represents a specific type determined at compile time when the function is invoked. This means, if in my code I do not call the function with the type `[]int` it won't have the function with the signature as `Print(stuff []int)` rather only the types which the function is called with are inferred and written with that specific types. ## Creating a Generic Slice We can even create a slice with a generic type and allow any valid processing on the elements or the slice as a whole. ```go package main import ( "fmt" ) func ForEach[T any](arr []T, f func(T)) { for _, v := range arr { f(v) } } func main() { strSlice := []string{"b", "e", "a"} ForEach(strSlice, func(v string) { fmt.Println(v) }) slice := []int{10, 25, 33, 42, 50} var evenSlice []int ForEach(slice, func(v int) { isEven := v%2 == 0 if isEven { evenSlice = append(evenSlice, v) } }) fmt.Println(evenSlice) } ``` ```bash $ go run main.go b e a [10 42 50] ``` [Go Playground Link](https://go.dev/play/p/tUwgbxnLc-1) The `ForEach` is a generic function that iterates over a slice of any type and calls a function on each element. Let's break it down: * `ForEach[T any]` declares this as a generic function that works on a slice of any type `T`. * `arr []T` is the slice of elements we want to iterate over. It can be a slice of any type, ints, strings, `T` in general, etc. So it is a generic slice parameter. * `f func(T)` is the function that will be called on each element. It takes a single parameter of type `T` which will be each element. So, this is a function parameter with a generic type as its parameter. In the body, we range over the slice arr: ```go for _, v := range arr { } ``` On each iteration, `v` will be the next element. The underscore ignores the index. We call the provided function `f`, passing the element `v`: `f(v)` So in summary: * It allows iterating over a slice of any type * This avoids having to know the specific slice type in the loop * The provided function `f` is called on each element * So it provides a reusable abstraction for processing slices generically. Now, we will discuss the example used in the main function. First, we create a slice of strings as `strSlice := []string{"b", "e", "a"}`. Then we call the generic `ForEach` function, passing the string slice and a function to handle each element. ```bash ForEach(strSlice, func(v string) { fmt.Println(v) }) ``` Here, the `func(v string)` is the invocation of the `ForEach` function with the typed string and the variable name as v. The `v` is the element of the slice, so inside the function body(this is an anonymous function), we call the `fmt.Println(v)`, which will print each string in the slice. ```go slice := []int{10, 25, 33, 42, 50} var evenSlice []int ForEach(slice, func(v int) { isEven := v%2 == 0 if isEven { evenSlice = append(evenSlice, v) } }) fmt.Println(evenSlice) ``` In the next example, we create a new slice of int as `slice := []int{10, 25, 33, 42, 50}`. Then we call the generic `ForEach` function again, passing the slice and a function to handle each element just as before, however, we are processing some things and then appending to an external slice. First, the `slice := []int{10, 25, 33, 42, 50}` is created with some numbers, we also initialize another slice as `evenSlice := []int{}` which is empty. Then we call the generic `ForEach` function again, passing the slice and a function to handle each [element.Here](http://element.Here), the ForEach is called with the `slice` slice and not the `evenSlice` slice, so we are going to access each element in the `slice` array. We first create a `isEven` boolean that checks if the element is even or odd by `v%2 == 0`. This expression will result in `true` if `v` is even and `false` otherwise. So, only if the `isEven` is true, we append the element `v` into the `evenSlice` slice. So, that's how generic slices can be handy for doing type-specific processing without writing functions for those individual types. This avoids needing to write type-specific functions for each slice type. NOTE: Make sure to only use generic functions with generic slice types with the appropriate and valid conditions and use it only when it looks legible to do so. ## Creating a Generic Map We can also create a generic map with the generic type of `map[K]T` where `K` is a generic type and `T` is the type of the key. ```go package main import ( "fmt" ) func GetValue[K comparable, V any](m map[K]V, key K, defaultVal V) V { if v, ok := m[key]; ok { return v } return defaultVal } func main() { serverStats := map[string]int{ "port": 8000, "pings": 47, "status": 1, "endpoints": 13, } v := GetValue(serverStats, "status", -1) fmt.Println(v) v = GetValue(serverStats, "cpu", 4) fmt.Println(v) } ``` ```bash $ go run main.go 1 4 ``` [Go Playground Link](https://go.dev/play/p/ludlh6UwKaD) GetValue is a generic function that takes three type parameters: The map itself, the key to find the value for, and a default value if the key doesn't exist. The `m` is a map with keys of type K and values of type V, the key is of type K, and defaultVal is of type V. So, we have two generics here, as the key and value need not be of the same type, hence we have distinct generics here. K has added a constraint of `comparable` and `V` as `any` type. The type constraint comparable restricts K to be a comparable type, and the type constraint `any` allows V to be any type. * Inside the function, we use the ok variable to check if the given `key` exists in the map `m`. * If the key is present in the map (ok is true), we retrieve the corresponding value from the map and return it as `m[key]` which is stored in the variable `v`. * If the key is not present in the map (ok is false), we return the provided `defaultVal`. So, this is how we can use any type of map to retrieve the value of a key, the data type of key and value could be anything. It allows us to retrieve a value from a map irrespective of the pair type in the map. NOTE: The type of `defaultVal` and the type of `v` should be the same since the map will need to have the value for the given key as the same type as defined in the map type(here `map[string]int` so `v` is `int` and so should be the `defaultVal`). Moving into the main function, we create a map of `[string]int` i.e. the key is of type `string` and the value of type `int`. The map `serverStats` has a few keys like `port`, `pings`, `endpoints`, and `status`. We call the `GetValue` method on the map `serverStats` with the key `status` and provide a default value of `-1`. The function will readily return `1` since the key is present in the provided map. However, if we try to access the key `cpu`, the key is not present and the value is returned as the `defaultVal` which we passed as `4`. So, this was a simple generic getter method on a map. We can get a value of a key in a map of any pair and even provide a default value if doesn't exist. However, it won't add it to the map, we will just return the value from the function that's it. We have to see that returned default value manually. We can make another function to get or set the value of a key in a map. The function will take in a reference to the map rather than a copy of the map, we can then use that reference to set the key with the provided default value. ```go package main import ( "fmt" ) func GetOrSetValue[K comparable, V any](m *map[K]V, key K, defaultVal V) V { // reference the original map ref := *m if v, ok := ref[key]; ok { return v } else { //mutate the original map ref[key] = defaultVal return defaultVal } } func main() { serverStats := map[string]int{ "port": 8000, "pings": 47, "status": 1, "endpoints": 13, } fmt.Println(serverStats) v := GetOrSetValue(&serverStats, "cpu", 4) fmt.Println(v) fmt.Println(serverStats) } ``` ```bash $ go run main.go map[endpoints:13 pings:47 port:8000 status:1] 4 map[cpu:4 endpoints:13 pings:47 port:8000 status:1] ``` [Go Playground Link](https://go.dev/play/p/fYtjFQRaPCb) In the above code, we take a reference of the map as `*map[K]V`, this will give access to the actual place(memory address where the map is located so we could mutate/change it). The rest of the parameters are kept as is, the key will be taken as it was before, and so will the `defaultVal`. The only difference is that we will set the `key` doesn't exist, we set the `ref[key]` to the `defaultVal` and return the `defaultVal`. For example, the `cpu` key is not present in the initial map `serverStats` so, when we call the `GetOrSetValue` with the reference to the map `serverStats`, key as `cpu` and the default value as `4`, the function returns `4` and the map is mutated with the key `cpu` with value `4`. The takeaway is you can even use references to access the original data and mutate it based on your needs. ## Generic Type in Struct We can define custom structs with generic type as the field values. The name of the struct is followed by the `[T any]` which is the type parameter to be used in the struct fields, again this type could have multiple types(since a struct can have many fields), not necessary want a single type parameter, you could have multiple type parameters bunched up just like we saw in the map example. ```go package main import ( "fmt" ) type Stack[T any] struct { items []T } func NewStack[T any]() *Stack[T] { return &Stack[T]{} } func (s *Stack[T]) Push(item T) { s.items = append(s.items, item) } func (s *Stack[T]) Pop() T { if len(s.items) == 0 { panic("Stack is empty") } index := len(s.items) - 1 item := s.items[index] s.items = s.items[:index] return item } func main() { intStack := NewStack[int]() intStack.Push(10) intStack.Push(20) intStack.Push(30) fmt.Println("Integer Stack") fmt.Println(intStack) intStack.Pop() intStack.Pop() fmt.Println(intStack) // without the NewStack method strStack := Stack[string]{} strStack.Push("c") strStack.Push("python") strStack.Push("mojo") fmt.Println("String Stack:") fmt.Println(strStack) strStack.Pop() fmt.Println(strStack) } ``` ```bash $ go run main.go Integer Stack &{[10 20 30]} &{[10]} String Stack: {[c python mojo]} {[c python]} ``` [Go Playground Link](https://go.dev/play/p/4t_P2mKtTZN) In this example, we have used the `Stack` example for doing a basic `Push` and `Pop` operation on the elements. Here the type of the underlying stack elements could be anything, hence the type parameter is defined for the `items` which is a list/slice of the type `T` as `[]T`. We have to specify the type before initializing the We have created the `NewStack` method, it is not needed, it could be just used as `Stack[int]{}` to initialize a empty stack with int type(here `int` could be any other type you wish). I have just created it so that it shows the abstraction that could be built upon in real applications. The `NewStack` simply replaces the `T` with the provided `type` in the initialization. The `Push` method is associated with the `Stack` struct, as we refer to the `*Stack[T]` indicating a reference to the Stack object with the type `T`. The method takes in the parameter `T` which would be the element to be added to the `Stack`. Since the function is tied to the Stack struct, we can simply mutate the underlying `items` by appending the provided value `item` in the parameter using `s.items = append(s.items, item)`. This appends the `item` to the underlying `items` list in the `Stack` object `s` Similarly, we can create `Pop` method as well, which will first check if the `Stack` is not empty(i.e. the s.items slice has a length greater than 0), then get the index of the last element using `len(s.items) - 1` and then slice the `items` at index `[:last_index]`. This will basically get you all the elements except the last. Before we remove the element from the slice, we also store the item in `item` variable and return it from the method. You could see the case of generics here, you could build complex data structures without creating a separate implementation for each type. ## Adding Constraints to Generics We can add constraints to the generics to restrict the type of the generic parameter. For example, we can add a constraint for the generic type to be a slice of a specific type or we have seen in the map example the `comparable` constraint. The `comparable` constraint is an interface that allows two instances of the same type to be compared i.e. support comparison operators like ==, &lt;, &gt;, !=, &gt;=, &lt;=, etc. It could be any numeric type like `int`, `float`, `uint` and variants, booleans, time duration, and any other struct that implements the `comparable` interface. ```go package main import ( "fmt" ) func FindIndex[T comparable](arr []T, value T) int { for i, v := range arr { if v == value { return i } } return -1 } func main() { strSlice := []string{"m", "e", "e", "t"} fmt.Println(FindIndex(strSlice, "e")) fmt.Println(FindIndex(strSlice, "t")) fmt.Println(FindIndex(strSlice, "a")) intSlice := []int{10, 25, 33, 42, 50} fmt.Println(FindIndex(intSlice, 33)) fmt.Println(FindIndex(intSlice, 90)) } ``` ```bash $ go run main.go 1 3 -1 2 -1 ``` [Go Playground Link](https://go.dev/play/p/fv9gzb8K4d7) In the above example, we have created the function `FindIndex` that takes in a generic slice, the type parameter `[T comparable]` indicates that the type used for calling this method needs to have a type that implements the comparable interface (for the elements of the slice). The method takes in two parameters, one the slice as `[]T` and the other the value to find the index for as type `T`. The method returns a type `int` since the index of the slice has to be an integer value. Inside the function body, we simply iterate over the slice `arr` and check if the element is equal to the provided value. If the element exists, we return that index, else we return `-1` As we can see we have run a couple of slices with the function `FindIndex` with types `int` and `string`. The method returns an index value if the element exists, else it returns `-1`. The `comparable` is a built-in type constraint. We could even define custom constraints as interfaces that implement the types of the particular type(s). Also, we could define custom constraints like numeric only, string only, etc. ```go package main import ( "fmt" ) type numeric interface { uint | uint8 | uint16 | uint32 | uint64 | int | int8 | int16 | int32 | int64 | float32 | float64 } func Sum[T numeric](nums []T) T { var s T for _, n := range nums { s += n } return s } func main() { intSlice := []int{10, 20, 30, 40, 50} fmt.Println(Sum(intSlice)) floatSlice := []float64{1.1, 2.2, 3.3, 4.4, 5.5} fmt.Println(Sum(floatSlice)) } ``` ```bash $ go run main.go 150 16.5 ``` [Go Playground Link](https://go.dev/play/p/_1eGl58qQ-2) In the above example, we have created the `numeric` constraints that allow the type `int`, `float` and their variants to be allowed in the numeric generic type. The function `Sum` is a generic function with the constraint of `numeric` type parameter. The method takes in the parameter as type `[]T` and returns the type as `T`. The method will simply iterate over the slice and return the sum of its elements. This will allow any numeric type which can be added and the sum can be obtained, so if we try to call the method with other types like `string` or `maps`, it won't work, and give an error: ```bash $ go run constraints.go # command-line-arguments scripts/generics/constraints.go:46:20: string does not satisfy numeric (string missing in uint | uint8 | uint16 | uint32 | uint64 | int | int8 | int16 | int32 | int64 | float32 | float64) shell returned 1 ``` So, we can use the constraint to restrict the type of the generic type parameter which will allow us to restrict the usage and avoid any unsafe type to be used in the generic function. Also, if we have a custom type with the base types, we need to use `~` before the type to accept it into the generic constraint. This will allow any approximate type to be allowed in the constraint. Let's say we are implementing a custom string type, for that to work with a constraint, it won't be satisfied in the constraint since its type is `CustomString` and not `string`. So to avoid this we use `~string` that would approximate the type and allow abstracted base types. ```go package main import ( "fmt" ) type string2 string type strings interface { ~string } func PrintEach[T strings](arr T) { for _, v := range arr { fmt.Printf("%c ", v) } } func main() { var s string2 s = "hello" PrintEach(s) } ``` ```bash $ go run main.go h e l l o ``` [Go Playground Link](https://go.dev/play/p/N-66A9C94ps) In the above example, we have used the type approximations in the type constraint `strings`, it allows any string type, whether a base `string` type or an abstract `string` type. If you try to remove the `~` in `~string`, it will result in the error that `string2 does not satisfy strings` interface. So, by adding `~` to the `string` type the abstract type `string2` can be satisfied in the generic constraint. That's it from the 29th part of the series, all the source code for the examples are linked in the GitHub on the [100 days of Golang](https://github.com/Mr-Destructive/100-days-of-golang/tree/main/scripts/generics) repository. ### References - [Generics in Go](https://bitfieldconsulting.com/golang/generics) - [Understanding generics in Go](https://blog.logrocket.com/understanding-generics-go-1-18/) ## Conclusion From this section of the series, we have covered the basics of generics in Golang. By using generics in functions, slices, maps, and structs, and adding constraints to them the fundamental usage of generics was covered. If you have any questions, feedback, or suggestions, please drop them in the comments/social handles or discuss them below. Thank you so much for reading. Happy Coding :) https://www.meetgor.com/posts/golang-date-time Introduction In the 28th post of the series, I will be exploring date and time handling in Golang. We will be covering the following topics: Date and Time parsi Mon, 19 Jun 2023 00:00:00 UTC ## Introduction In the 28th post of the series, I will be exploring date and time handling in Golang. We will be covering the following topics: * Date and Time parsing * Time Duration * Time and Date Arithmetic * Timezones * Sleep and Tickers This will cover most of the methods and properties used extensively in general use cases related to time and date operations. ## Time package The Golang standard library provides the time package to handle date and time-related operations. It has a lot of methods and constants to work and handle data related to time and dates. One of the fundamental methods to get time in golang is the [time.Now](http://time.Now) function, returns a time object representing the current time. The return value is a [time](https://pkg.go.dev/time#Time) object, which is the base struct that we can use to perform operations on top of it. ```go package main import ( "fmt" "time" ) func main(){ now := time.Now() fmt.Println(now) fmt.Printf("%T ", now) } ``` ```bash $ go run main.go 2023-06-18 15:18:53.977607431 +0530 IST m=+0.000050291 time.Time ``` ### Create a time/date type You can create a specific date by providing details like year, month, day, hour, minute, second, nanosecond, and time zone. The [time.Date](http://time.Date) method is used to create a time object. We will talk about location i.e. the timezone in a while. ```go package main import ( "fmt" "time" ) func main(){ sometime := time.Date(2020, 03, 25, 8, 5, 0, 0, time.UTC) fmt.Println(sometime) fmt.Printf("%T ", sometime) } ``` ```bash $ go run main.go 2020-03-25 08:05:00 +0000 UTC time.Time ``` The above example generates a custom time object as we parsed the year, month, day, etc to the function. The time zone is set to UTC, the method might also take in a custom timezone, but that is what we will be exploring further. We can also get some specific type from the time object using the methods that are available in the provided struct. For example, we can get the year from the time object by using the [time.Year()](https://pkg.go.dev/time#Time.Year), or month using [time.Month()](https://pkg.go.dev/time#Time.Month) and so on ```go package main import ( "fmt" "time" ) func main(){ somedate := time.Date(2020, 03, 25, 8, 5, 0, 0, time.FixedZone("UTC", 5)) fmt.Println(somedate) today := somedate.Day() fmt.Println(today) year := somedate.Year() fmt.Println(year) month := somedate.Month() fmt.Println(month) date := somedate.Day() fmt.Println(date) weekDay := somedate.Weekday() fmt.Println(weekDay) yearDay := somedate.YearDay() fmt.Println(yearDay) } ``` ```bash $ go run main.go 2020-03-25 08:05:00 +0000 UTC 25 2020 March 25 Wednesday 85 ``` In the above example, we have used the methods available in the time structure to get the specific components like a year, month, day, hour, etc. The data type for these methods is simple integer or strings as suitable to the format like int for date, year, and string for weekday, month, and timezone/location. ```go package main import ( "fmt" "time" ) func main(){ now := time.Now() fmt.Println(now.Format("Monday 01 January 2006 15:04:05")) // day month date hour:minutes:second timezone year fmt.Println(now.Format(time.UnixDate)) // day month date hour:minutes:second year fmt.Println(now.Format(time.ANSIC)) // day month date hour:minutes:second fmt.Println(now.Format(time.Stamp)) // day month date hour:minutes:second.milisecond fmt.Println(now.Format(time.StampMilli)) // day month date hour:minutes:second.microsecond fmt.Println(now.Format(time.StampMicro)) // day month date hour:minutes:second.nanosecond fmt.Println(now.Format(time.StampNano)) // day, date month year hour:minutes:second timezone fmt.Println(now.Format(time.RFC1123)) // day, date month year hour:minutes:second offset fmt.Println(now.Format(time.RFC1123Z)) // year-month-dayThour:minutes:second+-offset fmt.Println(now.Format(time.RFC3339)) // year-month-dayThour:minutes.nanosecond:second fmt.Println(now.Format(time.RFC3339Nano)) // date month year hour:minutes timezone fmt.Println(now.Format(time.RFC822)) // hour:minuteAM/PM fmt.Println(now.Format(time.Kitchen)) } ``` ```bash $ go run main.go Sunday 06 June 2023 20:09:09 Sun Jun 18 20:09:09 IST 2023 Sun Jun 18 20:09:09 2023 Jun 18 20:09:09 Jun 18 20:09:09.086 Jun 18 20:09:09.086565 Jun 18 20:09:09.086565975 Sun, 18 Jun 2023 20:09:09 IST Sun, 18 Jun 2023 20:09:09 +0530 2023-06-18T20:09:09+05:30 2023-06-18T20:09:09.086565975+05:30 18 Jun 23 20:09 IST 8:09PM ``` These are some of the time formats that are provided by the time package in golang. They all return a string, so it cannot be parsed and resolved into components again. You can definitely take in a custom timestamp, convert it into a time object and then use the appropriate time format for your needs. ## Parsing time object from a string We can use the [Parse](https://pkg.go.dev/time#Parse) function to parse a string into a time object. The method takes in two parameters, a date format, and the string to convert to both are parsed as a string. The method returns the parsed time or can give an error if the provided string is not in the mentioned format. ```go package main import ( "fmt" "time" ) func main(){ customDate := "2023-04-26" t, err := time.Parse("2006-01-02", customDate) if err != nil { fmt.Println(err) } fmt.Println(customDate) fmt.Println(t) customDate = "2023-0426" t, err = time.Parse("2006-01-02", customDate) if err != nil { fmt.Println(err) } fmt.Println(customDate) fmt.Println(t) } ``` ```bash $ go run main.go 2023-04-26 2023-04-26 00:00:00 +0000 UTC 2023-0426 parsing time "2023-0426" as "2006-01-02": cannot parse "26" as "-" 0001-01-01 00:00:00 +0000 UTC ``` In the above examples, we have parsed a time object from a string that looks like a date. We used the Parse method that takes in a string as the format to parse from and the string that we want to convert to. The format remains some specific date value like the `2006-01-02` which is fixed as a time to parse time in the time package. The method returns a time object which is parsed in the format, also it can return an error if the string is not in the provided format. We have used the second example that parsed a string as an invalid date format. ## Time Duration field The time duration field is used to represent the elapsed time between two Time objects as an integer 64-bit (in hours at the biggest unit and nanoseconds as the smallest unit to scale). ### Parse duration from string We can use the [ParseDuration](https://pkg.go.dev/time#ParseDuration) method to parse a duration like string into a time.Duration object. The duration object can have serialized fields like Hours, Minutes, and so on for further usage into the time processing. ```go package main import ( "fmt" "time" ) func main() { screentime, err := time.ParseDuration("6h30m") if err != nil { fmt.Println(err) return } fmt.Printf("%T ", screentime) fmt.Println(screentime.Hours()) fmt.Println(screentime.Minutes()) } ``` ```bash $ go run main.go time.Duration 6.5 390 ``` In this example, the time duration is 6 hours 30 minutes as indicated by `6h30m`. This is parsed as a time duration, the duration can accept formats like "ns", "us" (or "µs"), "ms", "s", "m", "h". We have used the `h` and `m` as short forms of hours and minutes respectively. Similarly, `s` can be used for seconds, `ms` for milliseconds, `us` or `µs` for microseconds, and `ns` for nanoseconds respectively. If you use another format it will result in errors, and yes! you will have to write err != nil { handle the error } syntax and debug the issue in the console (just saying) The [ParseDuration](https://pkg.go.dev/time#ParseDuration) method will return the [Duration](https://pkg.go.dev/time#Duration) object or will error out if the string is not in the required short duration formats. In this example, the duration is stored in the `screentime` variable, which has a few methods like `Hours`, `Minutes`, `Seconds`, and so on to extract the component time duration in that object. So, if we use `screentime.Hours()`, this will give us the total hours in that parsed duration, in this case, it is `6.5` hours. The return type is: * `float64` for [Hours](https://pkg.go.dev/time#Duration.Hours), [Minutes](https://pkg.go.dev/time#Duration.Minutes), and [Seconds](https://pkg.go.dev/time#Duration.Seconds) * `int64` for [Milliseconds](https://pkg.go.dev/time#Duration.Milliseconds), [Microseconds](https://pkg.go.dev/time#Duration.Microseconds), and [Nanoseconds](https://pkg.go.dev/time#Duration.Nanoseconds) This can be useful in Linux command line applications, where we can get the duration of the application running or execution speed, etc. It can be used to get the approximate number of hours or other metrics specific to the needs. The duration field is more useful for calculating the difference between two Time objects. Some methods like [Since](https://pkg.go.dev/time#Since), [Sub](https://pkg.go.dev/time#Time.Sub), are used to get the duration between the current time and other time objects and two Time objects respectively. ```go package main import ( "fmt" "time" ) func main() { newYear := time.Date(2023, 1, 1, 0, 0, 0, 0, time.UTC) // current time is 2023-06-18 15:27:12 +0000 UTC fmt.Println(time.Now().UTC()) fmt.Println(time.Since(newYear).Hours()) nextNewYear := time.Date(2024, 1, 1, 0, 0, 0, 0, time.UTC) fmt.Println(nextNewYear.Sub(newYear).Hours()) fmt.Println(nextNewYear.Sub(newYear).String()) } ``` ```bash $ go run main.go 2023-06-18 15:27:12 +0000 UTC 4047.4130468335657 8760 8760h0m0s ``` So as seen from the examples, the `Since` method calculates the difference between the current time and some other time object, and the `Sub` method calculates the difference between two Time objects. The current time is `18th June 2023` which is roughly near the middle of the year, so if we get the duration from the start of the year(1st January 2023), we get `4047.4` hours. Similarly, we can get a duration between a year, i.e. 1st January 2023 and 1st January 2024, which comes out to be `8760` hours. We can even use `Minutes`, `Seconds`, `Milliseconds`, etc. to get the duration in those units. ```go day := time.Hour * 24 fmt.Println(day) week := time.Hour * 24 * 7 fmt.Println(week) month := time.Hour * 30 * 24 * 7 fmt.Println(month) fifteenDays := day * 15 fmt.Println(fifteenDays) ``` ```bash $ go run main.go 24h0m0s 168h0m0s 5040h0m0s 360h0m0s ``` Here, we have created some custom units for calculating duration like a day which is 24 hours, that is obtained by multiplying an hour i.e. `time.Hour` by 24, and a week is 7 \* day, hence we do the calculation with precision in the duration structure. This can also be applied to `Minutes`, `Seconds`, and so on. ## Time Zones and Locations Time zones are associated with every time object in the time structure. They are represented as [Locations](https://pkg.go.dev/time#Location) structure. [List of time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) If not specified, the [`time.Now`](http://time.Now)`()` set the timezone as the Local time zone which is picked up from the system. It stores a list of timezone if the location uses daylight savings time, it picks the first timezone if daylight savings is not present. ```go package main import ( "fmt" "time" ) func main() { t := time.Now() fmt.Println(t) fmt.Println(t.Location()) newYorkTimeZone, err := time.LoadLocation("America/New_York") if err != nil { fmt.Println(err) } londonTimeZone, err := time.LoadLocation("Europe/London") if err != nil { fmt.Println(err) } newYorkTime := t.In(newYorkTimeZone) londonTime := t.In(londonTimeZone) //local time fmt.Println(t) // london time fmt.Println(londonTimeZone) fmt.Println(londonTime) // new york time fmt.Println(newYorkTimeZone) fmt.Println(newYorkTime) } ``` ```bash $ go run main.go 2023-06-19 13:58:13.232181805 +0530 IST m=+0.000044899 Local 2023-06-19 13:58:13.232181805 +0530 IST m=+0.000044899 Europe/London 2023-06-19 09:28:13.232181805 +0100 BST America/New_York 2023-06-19 04:28:13.232181805 -0400 EDT ``` The [LoadLocation](https://pkg.go.dev/time#LoadLocation) method is used to parse the timezone name as a string and returns the timezone/location object. It can return a `Location` object or an error in case the provided timezone name is not valid. We can use the [time.In](http://time.In)[(timezone)](https://pkg.go.dev/time#Time.In) to get the time in the specified timezone. In the above example, we have created a timezone/location object with the location name of the timezone as `Europe/London`. The timezone/location object has no exported/public properties or methods except the `String()` method used to get the string representation of the time stamp. However, this timezone is used to get time stamps in another timezone, for instance in this example, we have used my local timezone, i.e. `IST` or `Aisa/Kolkata` to get the current time, and used `In()` method to get time in `Europe/London` timezone. The `In()` method takes in a location object as a parameter and returns the `time.Time` object as the time in that provided location. ### Creating a custom time zone A location object in the time package is basically a name and an offset value, so we can construct our own custom time zone location object with those parameters with the [FixedZone](https://pkg.go.dev/time#FixedZone) method. The `FixedZone` method takes in two parameters, one as the name of the timezone as a string, and the other as the offset as int. The offset can be up to 290 years roughly, as it can hold 64 bits and this can only represent an integer value as time only this far. ```go package main import ( "fmt" "time" ) func main() { t := time.Now() fmt.Println(t) offset := int((4*time.Hour + 30*time.Minute + 30*time.Second).Seconds()) // (4*60 + 30*1 + 30*0.166) * 60 // (270 + 0.5) * 60 = 16230 fmt.Println(offset) fmt.Println(t.UTC()) customTimeZone := time.FixedZone("SOMETZ", offset) fmt.Println(customTimeZone) fmt.Println(t.In(customTimeZone)) } ``` ```bash $ go run main.go 2023-06-19 15:25:44.8624019 +0530 IST m=+0.000044899 16230 2023-06-19 09:55:44.8624019 +0000 UTC SOMETZ 2023-06-19 14:26:14.8624019 +0430 SOMETZ ``` So from the example, the offset is 16230 seconds, which is then constructed by adding hours, minutes, and seconds, this is just an example, it shows you can customize the hour, minute, and seconds components, but you can extend it with millisecond up to nanosecond but that is a very niche case and might not be broadly used. To get an offset of `4h3030s`, we have used arithmetic to get the number of hours, minutes, and seconds. ## Arithmetic on Time We can perform addition(going ahead) and subtraction(going back) of time in the `time.Time` object. There are also some comparison operations that can be performed using the method provided in the time struct. ### Add Time and Date We can add a date to the `time.Time` object using the [Add](https://pkg.go.dev/time#Time.Add) method, this method takes in an integer value of `Hours`, `Minute`, `Second`, and so on to add the value of those units to the existing time object as a copy. This method can be chained to add multiple units of different types for instance hours, minutes, or seconds in a precise way to add the time. There is also a [AddDate](https://pkg.go.dev/time#Time.AddDate) method which takes in 3 parameters as a number of years, months, and days. We can add `x` years to the existing date by passing `AddDate(x, 0, 0)` to the time object we want to add to. These two method can be chained together to go from a scale of year to all the way to nanoseconds precision in adding the time. ```go package main import ( "fmt" "time" ) func main() { t := time.Now() fmt.Println(t) afterOneHour := t.Add(time.Hour * 1) fmt.Println(afterOneHour) afterOneDayTwoHours30minutes := t.AddDate(0, 0, 1).Add(time.Hour * 2).Add(time.Minute * 30) fmt.Println(afterOneDayTwoHours30minutes) afterThreeMonths15days := t.AddDate(0, 3, 15) fmt.Println(afterThreeMonths15days) } ``` ```bash $ go run main.go 2023-06-19 15:58:32.893246798 +0530 IST m=+0.000042833 2023-06-19 16:58:32.893246798 +0530 IST m=+3600.000042833 2023-06-20 18:28:32.893246798 +0530 IST 2023-10-04 15:58:32.893246798 +0530 IST ``` Here, we can see, we have taken a simple example, for adding an hour to the existing time by saying `t.Add(time.Hour)`, this can be multiplied by the number of hours to add to, in this case, it was just one so we simply multiply with one. In the next example, we have chained the `AddDate` and `Add` methods to get the time after 1 day, 2 hours, and 30 minutes. The `AddDate` method is passed with `(0, 0, 1)` indicating a single day, then we tune the hours with 2 and minutes with 30 to get the desired time. In the last example, we have added the time by 3 months and 15 days, bypassing the `AddDate` method with `(0, 3, 15)`. Time can even be subtracted or we can get behind the specified time object using the negative number in the `Add` method. Instead of saying `.Add(1 * time.Hour)` to go one hour ahead of the parsed time, we can say `.Add(-1 * time.Hour)` to go one hour behind the current time. ```go package main import ( "fmt" "time" ) func main() { t := time.Now() fmt.Println(t) oneHourBack := t.Add(-1 * time.Hour) fmt.Println(oneHourBack) beforeOneYearTwoMonths := t.AddDate(-1, -2, 0) fmt.Println(beforeOneYearTwoMonths) } ``` ```bash $ go run main.go 2023-06-19 16:23:26.992724868 +0530 IST m=+0.000044899 2023-06-19 15:23:26.992724868 +0530 IST m=-3599.999957644 2022-04-19 16:23:26.992724868 +0530 IST ``` In this example, we have used the `Add` and `AddDate` methods with negative numbers to go back in time. In the first example, we have subtracted 1 hour from the existing time object, in the second example, we have used -1 to go back a year and -2 for going back 2 months from the current time object, hence we pass `.AddDate(-1, -2, 0)` in the example. ### Comparing Time We can use `After`, `Before`, `Equal`, or `Compare` methods to compare the `Time` object in Golang. The method returns true or false depending if the time comes after, before, or is equal, the compare method returns -1 if it is before, 0 if two times are equal, and 1 if the time is after the other. ```go package main import ( "fmt" "time" ) func main() { t := time.Now() fmt.Println(t) afterOneHour := t.Add(time.Hour * 1) fmt.Println(afterOneHour) isNowAfter := t.After(afterOneHour) isOneAfter := afterOneHour.After(t) fmt.Println(isNowAfter) fmt.Println(isOneAfter) isNowBefore := t.Before(afterOneHour) isOneBefore := afterOneHour.Before(t) fmt.Println(isNowBefore) fmt.Println(isOneBefore) } ``` ```bash $ go run main.go 2023-06-19 16:38:25.785629649 +0530 IST m=+0.000051616 2023-06-19 17:38:25.785629649 +0530 IST m=+3600.000051616 // is now after one hour? false // is one hour after now? true // is now before after one hour? true // is after one hour before now? false ``` These are some ridiculous examples, but it could be any date comparison like billing periods, subscription due, etc. We can compare dates with [After](https://pkg.go.dev/time#Time.After), [Before](https://pkg.go.dev/time#Time.Before), and [Equal](https://pkg.go.dev/time#Time.Equal) methods provided in the time package. ```go package main import ( "fmt" "time" ) func main() { t := time.Now() fmt.Println(t) afterOneHour := t.Add(time.Hour * 1) fmt.Println(afterOneHour) isNowEqual := t.Equal(afterOneHour) isEqual := time.Now().Equal(t) fmt.Println(isNowEqual) fmt.Println(isEqual) londonTimeZone, err := time.LoadLocation("Europe/London") if err != nil { fmt.Println(err) } londonTime := t.In(londonTimeZone) fmt.Println(t) fmt.Println(londonTime) fmt.Println(t.Equal(londonTime)) } ``` ```bash $ go run main.go 2023-06-19 16:38:25.785629649 +0530 IST m=+0.000051616 2023-06-19 17:38:25.785629649 +0530 IST m=+3600.000051616 // is now equal to one hour after now? false // is t equal to now? false 2023-06-19 16:49:25.509421027 +0530 IST m=+0.000040200 2023-06-19 12:19:25.509421027 +0100 BST // is london time equal to now in local? true ``` Here in the examples, we have checked if two Time objects are equal or not, we have checked if a current time is equal to one hour after time, and it returns false rightly so. In the next example, we have created a new time object as [`time.Now`](http://time.Now), and then compared with the previous now, there will be a difference in the seconds of the initialization and hence it gives false as the return value. In the next example, we take the current time and the current time in the London location as the `BST` timezone, this gets the result as `true` because the time is the same despite being in different time zones. ## Sleep Time We can even sleep for a certain duration of time, with the [Sleep](https://pkg.go.dev/time#Sleep) method in the time package, this results in halting the execution of the program. ```go package main import ( "fmt" "time" ) func main() { t1 := time.Now() time.Sleep(time.Second * 3) t2 := time.Now() duration := t2.Sub(t1) fmt.Println(duration) } ``` ```bash $ go run main.go 3.000235856s ``` In this example, we have used the `Sleep()` method with a duration of 3 seconds which will halt the execution of the program (or the goroutine) for that duration. We also printed the duration for which the flow was halted by taking a timestamp before and after the sleep method was called and then taking the difference between the two-time stamps. This rightly gives us a value of three seconds which is the duration for which we tried to call the `Sleep` method. ## Tickers Tickers are basically like a clock that ticks at regular intervals. It's a mechanism provided by the package to execute code repeatedly at fixed time intervals. The [Ticker](https://pkg.go.dev/time#Ticker) is a type and it has a few methods associated with it for handling these ticks. The [NewTicker](https://pkg.go.dev/time#NewTicker) method is used to create a ticker with the parameter as the amount of time to repeat the tick i.e. the frequency of its ticking. It will only stop when the [.Stop](https://pkg.go.dev/time#Ticker.Stop) method is called with that ticker object. ```go package main import ( "fmt" "time" ) func main() { ticker := time.NewTicker(time.Second * 2) counter := 0 for { select { case <-ticker.C: // api calls, call to database after specific time intervals counter++ fmt.Println("Tick", counter) } if counter == 5 { ticker.Stop() return } } } ``` ```bash $ go run main.go Tick 1 Tick 2 Tick 3 Tick 4 Tick 5 ``` In the example above, we have created a ticker with a frequency of 2 seconds, we initialize a counter to 0, we run an infinite condition for loop, and the ticker receives a value when the time is for ticking via the channel `.C`, this is used to check if it has ticked, and we enter the flow for the execution of any logic/code. We internally increment the counter and when the counter is 5 we stop the ticker and break out of the loop with either a `break` or `return`. So, that is how tickers can be used, this can be used in sending requests to APIs that have rate limiting, so we make sure we don't flood the API and maintain the rate of hitting the API only after specific intervals with tickers. That's it from the 28th part of the series, all the source code for the examples are linked in the GitHub on the [100 days of Golang](https://github.com/Mr-Destructive/100-days-of-golang/tree/main/scripts/date-time) repository. ## Conclusion So, in conclusion, we have seen how to use the time package in Go. We covered time and date parsing, timezone, date comparison, sleep, and tickers. This would have given a good overview of the time package in golang. If you have any questions, feedback, or suggestions feel free to drop them in the comments section, or on the social handles. Thank you for reading. Happy Coding :) https://www.meetgor.com/posts/golang-random-numbers Introduction In the 27th post of the series, we will be looking into random number generation in golang. We will be exploring how to create a random number, gen Sun, 14 May 2023 00:00:00 UTC ## Introduction In the 27th post of the series, we will be looking into random number generation in golang. We will be exploring how to create a random number, generate random numbers within a range, shuffle slices/arrays, and generate random strings, float, and bytes. There are two types of random number generation processes in software pseudo-random numbers and cryptographically secure pseudo-random number generation. The [math/rand](https://pkg.go.dev/math/rand) package in Golang provides a number of functions for generating pseudorandom numbers. These functions are suitable for a variety of applications, such as games, simulations, and to some extent in cryptography. The [crypto/rand](https://pkg.go.dev/crypto/rand) package in Golang generates cryptographically secure pseudorandom numbers. This means that the numbers are generated in a way that is very difficult to predict or reproduce. However, they are not truly random, as they are generated by a computer algorithm. ## Creating a Random Source We need to first create a source/seed to generate a random number. If we do not add a new source each time we run the program or generate a random number, it will pick up the same source leading to the generation of the same pattern of the random numbers. We use the [rand.NewSource](https://pkg.go.dev/math/rand#NewSource) method to generate a new source of the random number, by initializing it with the current time in nanoseconds. Further, we need to create a [Rand](https://pkg.go.dev/math/rand#Rand) object for accessing the methods and properties attached to the struct type in the rand package for generating random numbers. ```go package main import ( "fmt" "math/rand" "time" ) func main() { source := rand.NewSource(time.Now().UnixNano()) rand_source := rand.New(source) for i := 0; i < 5; i++ { rand_num := rand_source.Int() fmt.Println(rand_num) } } ``` ```bash $ go run rand_source.go 2651653079934875120 5510445616427469234 3817011159463415912 5708754224362255659 7511308401304127761 ``` In the above example, we use the `NewSource` and the `New` method to generate the `Source` and the `Rand` object respectively. Further, for demonstration, we use the [Int](https://pkg.go.dev/math/rand#Rand.Int)method to generate a 64-bit random integer 5 times with a for loop. As we can see it generates a 5 random number, we will see why we need the random source initialized to the current time in the upcoming section. ## Generating Random Numbers Random number as the name suggest are used to get an unpredictable number, however, using software we can only mimic the actual randomness. The process is called pseudo-random number generation. There is a particular pattern in the numbers, however, it is sufficient for trivial tasks in games, and simulations to some extent. However, actual cryptographic random numbers should be used for security tasks, crypto arithmetic, and other sensitive tasks. Golang provides a built-in package for both generating pseudo-random numbers called math/rand and cryptographic numbers with crypto/rand packages. This package contains a number of functions for generating random integers, floats, and strings. ### Random Numbers To simply generate a random number we can use the `rand.Int` method from the [match/rand](https://pkg.go.dev/math/rand#Int) package to get a single random integer. ```go package main import ( "fmt" "math/rand" ) func main() { fmt.Println(rand.Int()) num := rand.Int() fmt.Printf("%d", num) } ``` ```bash $ go run main.go 5577006791947779410 int 8674665223082153551 $ go run main.go 5577006791947779410 int 8674665223082153551 ``` In the above code, we have generated a couple of random numbers with the `rand.Int` method. The method returns a random 64-bit integer. If you run the program a few times, you can see the numbers are the same, so how exactly are they random? They are not random yet, we need to create a new seed/source each time we run the program in order to generate a new pattern of digits each time to generate a pseudo-random number. ```go package main import ( "fmt" "math/rand" "time" ) func main() { r := rand.New(rand.NewSoure(time.Now().UnixNano())) fmt.Println(r.Int()) num := r.Int() fmt.Printf("%d", num) } ``` ```bash $ go run main.go 5577006791947779410 int 8674665223082153551 7524437893560534176 int 5023070266853767708 $ go run main.go 5577006791947779410 int 8674665223082153551 8935404877937414882 int 209789266380754935 ``` No, we can see that after the `rand.New(rand.NewSource(time.Now().UnixNano()))` function call, the number generated is random each time we run the program. This is because we initialize the source of the random number generator package to the current time in nanoseconds. ### Random Numbers in a Range The above numbers are too big, what if we want the random numbers to be in a specific range? This is quite a common thing to do, and hence there is a function like [rand.Intn](https://pkg.go.dev/math/rand#Intn) where we can specify the bound to which the function should generate the random numbers. ```go package main import ( "fmt" "math/rand" "time" ) func main() { r := rand.New(rand.NewSoure(time.Now().UnixNano())) for i := 0; i < 10; i++ { // generate an integer between 0 and 5 dice_throw := r.Intn(6) // Move the Offset of 0 fmt.Println(dice_throw + 1) } } ``` ```bash $ go run main.go 1 2 5 6 6 3 6 1 4 2 ``` In the above code, we have used the `r.Intn(upper_range int)` method to generate a random number between 0 and the provided range, so if we give a parameter to the method `r.Intn(6)`, it would generate the numbers between 0 and 5. so the range is not inclusive 0, 6). Thereby the numbers generated will be either 0, 1, 2, 3, 4, or 5. So to remove the offset of 0, we add 1. Hence we get some pseudo-random numbers between 1 and 6. I have used a for loop that generates 10 such numbers. ### Cryptographic Random Numbers The above method was a pseudo-random number generator, however, for more robust random number generations, we can use the [crypto/rand](https://pkg.go.dev/crypto/rand) package that is more secure and powerful for complex operations. ```go package main import ( crypto_rand "crypto/rand" "fmt" "math/big" ) func Handle_error(err error) { if err != nil { panic(err) } } func main() { // Cryptographic random numbers var max *big.Int = big.NewInt(6) // big is a package for high-precision arithmetic for i := 0; i < 10; i++ { crypt_rand_num, err := crypto_rand.Int(crypto_rand.Reader, max) Handle_error(err) // Move the Offset of 0 by adding 1 crypt_num := crypt_rand_num.Add(crypt_rand_num, big.NewInt(1)) fmt.Println(crypt_num) } } ``` ```bash $ go run main.go 3 5 5 1 5 5 4 2 3 6 ``` In the above example, we have used the [math/big](https://pkg.go.dev/math/big) package to store the random number generated by the [Int](https://pkg.go.dev/crypto/rand#Int) method in the `crypto/rand` package. We create a new integer from the `big` package. The [NewInt](https://pkg.go.dev/math/big#NewInt) function returns a pointer to the integer. So, we parse the integer 6 which will create a memory location storing 6 as a [big.Int](https://pkg.go.dev/math/big#Int) type. We use the max variable name as it denotes the maximum number to be generated in the next step. Then we can use the [crypto/rand](https://pkg.go.dev/crypto/rand) package to generate cryptographic random numbers. The package has [Int](https://pkg.go.dev/crypto/rand#Int) method to generate a `big.Int` type of number in a given range. However, it also takes in a [Reader](https://pkg.go.dev/crypto/rand#pkg-variables) object that is global in the package used as a shared instance of a cryptographically secure random number generator. This means it can be used as the platform's default random number generator in the program. So, the `crypto_rand.Int` method takes in two parameters, the `Reader` object which will be the platform-specific random number generator/api, and the next parameter is the max range to generate the random number. So, this method returns a `big.Int` type. This is the cryptographic random number. However, we have the range from 0 to 5 again, so we just add 1 to the `big.Int` type by using the [Add](https://pkg.go.dev/math/big#Int.Add) method associated to the `big.Int` type as `crypto_rand_num` variable and then parse two parameters as x and y, i.e. the two numbers to add. So, we just pass the `crypto_rand_num` and a new integer `1`. This adds the numbers and we store it in the `crypto_num` variable. This is how we generate a cryptographic random number between 1 and 6. ### Random Float We can even generate random types like float. There are a quite a few variations like [Float32](https://pkg.go.dev/math/rand#Float32), [Float64](https://pkg.go.dev/math/rand#Float64), [ExpFloat64](https://pkg.go.dev/math/rand#ExpFloat64), and [NormFloat64](https://pkg.go.dev/math/rand#NormFloat64). ```go package main import ( "fmt" "math/rand" "time" ) func main() { r := rand.New(rand.NewSoure(time.Now().UnixNano())) rand_float32 := r.Float32() fmt.Println(rand_float32) rand_float64 := r.Float64() fmt.Println(rand_float64) rand_exp_float := r.ExpFloat64() fmt.Println(rand_exp_float) rand_norm_float := r.NormFloat64() fmt.Println(rand_norm_float) for i := 0; i < 5; i++ { rand_float := r.Float32() fmt.Println(rand_float) } ``` ```bash $ go run main.go 0.08891088 0.9218221078616824 1.8237338579299396 -0.30238778160373464 0.65474856 0.65964687 0.39930198 0.8043338 0.17894344 ``` We have used 4 types for generating a random float. We have two variations of the float depending on the size, 32 or 64-bit number. We get 32-bit and 64-bit random floats from the `r.Float32` and `r.Float64` respectively. The `r.ExpFloat64` function returns an exponentially distributed float64 with a range from 0 to +math.MaxFloat64 with rate parameter and mean as 1. If you want to change the distribution's rate parameter, it can be done by dividing the number by the desired rate parameter. The `r.NormFlaot64` function returns a normally distributed float64 with a range from -math.MaxFloat64 to +math.MaxFloat64 with mean as 0 and standard deviation as 1. This can also be changed by multiplying the generated number by the desired standard deviation and then adding the desired mean. ## Generating Random Strings We can generate a random string of a specific length. We can generate a random number between 0 and 25 and then add 97 for lowercase ASCII characters and add 65 for uppercase characters. So, we generate a random number between 97 and 122 which can be cast to string/rune to get the string equivalent of the numbers. This get's us a single character which would be random and thereby we use for loop to generate a fixed length random string for upper case and lower case characters similarly. ```go package main import ( "fmt" "math/rand" "time" ) func main() { r := rand.New(rand.NewSoure(time.Now().UnixNano())) //Random string randomLowerCase := make([]rune, 6) randomUpperCase := make([]rune, 6) for i := range randomLowerCase { randomLowerCase[i] = rune(r.Intn(26) + 97) randomUpperCase[i] = rune(r.Intn(26) + 65) } randomLowerCaseStr := string(randomLowerCase) randomUpperCaseStr := string(randomUpperCase) fmt.Println(randomLowerCase) fmt.Println(randomLowerCaseStr) fmt.Println(randomUpperCase) fmt.Println(randomUpperCaseStr) } ``` ```bash $ go run main.go [100 113 122 97 107 101] dqzake [86 81 88 76 66 74] VQXLBJ $ go run main.go [116 115 120 97 100 111] tsxado [80 74 66 83 77 66] PJBSMB ``` We first create an empty or 0 initialized rune slice with length 6, it can be any length as per your requirement. Then we create a for loop iterating over that slice and set each rune to the `r.Intn` method with range 26 and add 97 for lower case letters and add 65 for upper case letters. This generates an integer between 97 to 122 that is typecast to a rune to represent the slice. Similarly, it generates an integer between 65 and 90 which is typecast to rune. ## Shuffling Arrays We can use the [rand.Shuffle](https://pkg.go.dev/math/rand#Shuffle) and the [rand.Perm](https://pkg.go.dev/math/rand#Perm) to shuffle and create a random permutation of a particular list/slice of elements. ```go package main import ( "fmt" "math/rand" "time" ) func main() { r := rand.New(rand.NewSoure(time.Now().UnixNano())) fmt.Println(r.Perm(10)) arr := []int{1, 2, 3, 4, 5, 6} fmt.Println("Before shuffle:", arr) r.Shuffle(len(arr), func(i, j int) { arr[i], arr[j] = arr[j], arr[i] }) fmt.Println("After shuffle:", arr) } ``` ```bash $ go run main.go [8 1 9 3 7 2 0 6 5 4] Before shuffle: [1 2 3 4 5 6] After shuffle: [4 6 5 3 2 1] ``` In the above example, we have used the `rand.Perm` method to create a permutation of the n number passed as a parameter. So, it would generate a permutation of numbers from 0 to 9 if we pass 10 as the parameter to the method. It would return a slice of int. We also have used the `rand.Shuffle` method to shuffle an already existing slice of elements. This is not restricted to int, it can be any type of slice. The method takes in two parameters, the length of the slice/array and the swap function which is an anonymous function. In the example, I have created an arr slice with 6 elements, it could be any number, for demonstration, I have initialized the slice with 1 to 6 numbers. The `rand.Shuffle` method has been parsed with the length of the `arr` as `len(arr)` and an anonymous function that takes in two integers as the indices of the array and inside the function, we swap the elements of the array. This can be modified as per the requirement, but this is the base swap function for the shuffle method. So, in the output, we print the shuffled array, which now looks like a random array of numbers. We can use the `Perm` method to generate a random list of indices of an array, and then assign the index to the string, to generate a random shuffled string. ```go package main import ( "fmt" "math/rand" "time" ) func main() { r := rand.New(rand.NewSoure(time.Now().UnixNano())) letters := "abcdefghijklmnopqrstuvwxyz" shuffled := r.Perm(len(letters)) result := make([]byte, len(letters)) for i, randIndex := range shuffled { result[i] = letters[randIndex] } rand_str := string(result) fmt.Println(rand_str) // random string of length 10 fmt.Println(rand_str[:10] } ``` ```bash $ go run main.go yeinvkdbfqomacrzhtswgxulpj yeinvkdbfq $ go run main.go tvakbgnjprwiofquxlzecdshym tvakbgnjpr ``` In the above example, we have first created the string with all the alphabets and then created a random permutation with the length of that array i.e. 26. This would create a random permutation of numbers from 0 to 25. This is now a list of numbers, which can be used to assign the index of the string, to make it feel like a shuffled string. We create a for loop for the string iteration and assign the index with the random operation array index. This will basically jumble the order of the elements/characters in the string. We can then truncate or slice the string to any length as per the requirement. ## Random Read Bytes There is another way to generate a slice of bytes/string with the [Read](https://pkg.go.dev/math/rand#Rand.Read) method. We have used the `Read` method in the cryptographic random number generation part. The Read method generates a random byte for the given length of a slice of bytes. ```go package main import ( crypto_rand "crypto/rand" "fmt" "math/big" "math/rand" "time" ) func Handle_error(err error) { if err != nil { panic(err) } } func main() { r := rand.New(rand.NewSoure(time.Now().UnixNano())) rand_byte := make([]byte, 5) fmt.Println(rand_byte) _, err = r.Read(rand_byte) Handle_error(err) fmt.Println(rand_byte) fmt.Printf("%c ", rand_byte) crypt_rand_byte := make([]byte, 5) fmt.Println(crypt_rand_byte) _, err = crypto_rand.Read(crypt_rand_byte) Handle_error(err) fmt.Println(crypt_rand_byte) fmt.Printf("%c ", crypto_rand_byte) } ``` ```bash $ go run main.go [0 0 0 0 0] [88 53 113 116 251] [X 5 q t û] [0 0 0 0 0] [37 90 42 93 96] [% Z * ] `] ``` We have demonstrated usage of both the packages i.e. math/rand and crypto/rand for the generation of random bytes. In the example above, we initialize a slice of byte `rand_byte` and use the `Read` method that will take in the slice of byte as the parameter and return two things, the number of bytes it read and the error object if there is any or nil. So, we do not care how many bytes it read right now, so we do the `_` for the read bytes. It mutates/modifies the byte slice and the slice elements are then random byte values. We can print the slice of byte as a string with `%s`, or each character in the bytes using the `%c` format specifier. So, the generated bytes are between 0 and 255, which include Unicode and ASCII characters. Similarly, for the crypto/rand package, we create a slice of bytes with size 5 and use the crypto/rand package provided [Read](https://pkg.go.dev/crypto/rand#Read) method directly to modify the slice of bytes to random bytes. That's it from the 27th part of the series, all the source code for the examples are linked in the GitHub on the [100 days of Golang](https://github.com/Mr-Destructive/100-days-of-golang/tree/main/scripts/random-num) repository. ## Conclusion From this part of the series, we were able to explore the random number generation in golang, packages like math/rand, mah/big, and crypto/rand were used for the examples and generation of random numbers, cryptographic secure random numbers, random strings and shuffling of arrays. So, hopefully, the article might have found useful to you, if you have any queries, questions, feedback, or mistakes in the article, you can let me know in the discussion or on my social handles. Happy Coding :) https://www.meetgor.com/posts/golang-regex Introduction In this 26th part of the series, we will be covering the basics of using regular expressions in golang. This article will cover the basic operation Wed, 22 Mar 2023 00:00:00 UTC ## Introduction In this 26th part of the series, we will be covering the basics of using regular expressions in golang. This article will cover the basic operations like matching, finding, replacing, and sub-matches in a regular expression pattern from string source or file content. This will have examples for each of the concepts and similar variants will follow the same ideology in self-exploring the syntax. ## Regex in golang So, let's start with what are regular expressions. > Regular expressions are basic building blocks for searching, pattern matching, and manipulation from a source of text using computational logic. This is not the formal definition, but have written it in words for my understanding of regular expressions till now. Might not be accurate, but makes sense to me after I had played and explored it (not fully). So regular expressions use some pattern-matching techniques using basic logic operators like concatenation, quantifiers, etc. These relate to the study of the theory of computation quite closely but you don't need to get into too much theory in order to understand the working of regular expressions. However, it won't harm you if you are curious about it and want to explore it further. Some resources to learn the fundamentals of regular expressions: * [Regular Expressions LMU notes](https://cs.lmu.edu/~ray/notes/regex/) * [RegexOne](https://regexone.com/) ## Regexp package We will be using the [regexp](https://pkg.go.dev/regexp) package in the golang standard library to get some quick and important methods for quick and neat pattern matching and searching. It provides a `Regexp` type and a lot of methods on top of it to perform matching, finding, replacing, and sub-matches in the source text. This package also supports two types of methods serving different purposes and use cases for string and slice of bytes, this can be useful for reading from a buffer, file, etc., and also flexible enough to search for a simple string. ## Matching Patterns One of the fundamental aspects of the regular expression is to check if a particular pattern is present or not in a source string. The `regexp` package provides some methods like [Match](https://pkg.go.dev/regexp#Match), [MatchString](https://pkg.go.dev/regexp#MatchString) methods on a slice of bytes and string respectively from the pattern string. ### Matching Strings The basic operations with regex or regular expression can be performed to compare and match if the pattern matches a given string. In golang, the [regexp](https://pkg.go.dev/regexp) package provides a few functions to simply match expressions with strings or text. One of the easy-to-understand ones include the [MtachString](https://pkg.go.dev/regexp#MatchString), and [Match](https://pkg.go.dev/regexp#Match) methods. ```go package main import ( "log" "regexp" ) func log_error(err error) { if err != nil { log.Fatal(err) } } func main() { str := "gophers of the goland" is_matching, err := regexp.MatchString("go", str) log_error(err) log.Println(is_matching) is_matching, err := regexp.MatchString("no", str) log_error(err) log.Println(is_matching) } ``` ```console $ go run main.go true false ``` In the above code, we have used the `MatchString` method that takes in two parameters string/pattern to find, and the source string. The function returns a boolean as `true` or `false` if the pattern is present in the source string, also it might return an error if the pattern(first parameter) is parsed as an incorrect regular expression. So, we can clearly see, the string `go` is present in the string `gophers of the goland` and the string `no` is not a substring. We also have `Match` method which is a more general version of `MatchString` it excepts a slice of byte rather than a string as the source string. The first parameter is still a string, but the second parameter is a slice of bytes. ```go is_matching, err = regexp.Match(`.*land`, []byte("goland is a land of gophers")) log_error(err) log.Println(is_matching) ``` ```console $ go run main.go true ``` We can use the `Match` method to simply parse a slice of bytes to use as the source text to check the pattern. ```go package main import ( "log" "os" "regexp" ) func log_error(err error) { if err != nil { log.Fatal(err) } } func main() { file_content, err := os.ReadFile("temp.txt") log_error(err) is_matching, err = regexp.Match(`memory`, file_content) log_error(err) log.Println(is_matching) is_matching, err = regexp.Match(`text `, file_content) log_error(err) log.Println(is_matching) } ``` ```bash # temp.txt One of the gophers used a slice, the other one used a arrays. Some gophers were idle in the memory. ``` ```console $ go run main.go true false ``` We can now even parse the contents of a file as a slice of bytes. So, it would be really nice to compare and check for a string pattern in a file quickly. Here in the above example, we have checked if `memory` is present in the text which it is, and in the second call, we check if `text` string is present anywhere in the file content which it is not. ## Find Patterns We can even use regular expressions for searching text or string with the struct/type [Regexp](https://pkg.go.dev/regexp#Regexp) provided by golang's regexp. We can create a regular expression and use other functions like `MatchString`, `Match`, and others that we will explore to match or find a pattern in the text. ### Find String from RegEx We can get a slice of byte from the `FindAll` method which will take in a slice of byte, the second parameter as -1 for all matches. The function returns a slice of slice of byte with the byte representation of the matched string in the source text. ```bash exp, err := regexp.Compile(`�\d{5}(?:[-\s]\d{4})?�`) log_error(err) pincode_file, err := os.ReadFile("pincode.txt") log_error(err) match := exp.FindAll(pincode_file, -1) log.Println(match) ``` ```bash # pincode.txt Pincode: 12345-1234 City, 40084 State 123 ``` ```console $ go run main.go [[49 50 51 52 53 45 49 50 51 52] [52 48 48 56 52]] ``` In the above example, we have used the [Compile](https://pkg.go.dev/regexp#Compile) method to create a regular expression and [FindAll](https://pkg.go.dev/regexp#Regexp.FindAll) to get all the occurrences of the matching patterns in the text. We have again read the contents from the file. In this example, the `exp` is a regular expression for a postal code, which can either have 5 digit or 5digits-4digit combination. We read the file `pincode.txt` as a slice of bytes and use the `FindAll` method. The FindAll method takes in the parameter as a slice of byte and the integer as the number of occurrences to search. If we use a negative number it will include all the matches. We search for the pin code in the file and the funciton returns a list of bytes that match the regular expression in the provided object `exp`. Finally, we get the result as `12345-1234` and `40084` which are present in the file. It doesn't match the number `123` which is not a valid match for the given regular expression. There is also a version of `FindAll` as `FindAllString` which will take in a string as the text source and return a slice of strings. ```go matches := exp.FindAllString(string(pincode_file), -1) log.Println(matches) ``` ```console $ go run main.go ["12345-1234" "40084"] ``` So, the `FindAllString` would return a slice of strings of the matches in the text. ### Find the Index of String from RegEx We can even get the start and end index of the matched string in the text using the [FindIndex](https://pkg.go.dev/regexp#Regexp.FindIndex) and [FindAllIndex](https://pkg.go.dev/regexp#Regexp.FindAllIndex) methods to get the indexes of the match, or all the matches of the file content. ```go exp, err := regexp.Compile(`�\d{5}(?:[-\s]\d{4})?�`) log_error(err) pincode_file, err := os.ReadFile("pincode.txt") log_error(err) match_index := exp.FindIndex(pincode_file) log.Printf("%T ", match_index) log.Println(match_index) ``` ```console $ go run main.go []int [9 19] ``` The above code is using the `FindIndex` method to get the indexes of the first match of the regular expression. The output of the code gives a single slice of an integer with length two as the start and last index of the matched string in the text file. So, here, the `9` represents the position(index) of the first character match in the string, and `19` is the last index of the matched string. ```bash Pincode: 12345-1234 01234567890123456789 Assume 0 as 10 after 9, 11, and so on It starts from the 9th character as `1` and ends at the `4` character at position 18 but it returns the end position + 1 for the ease of slicing City, 40084 012345678901 State 123 234567890 ``` The character at 9 and 18 are the first and the last character position/index of the source string, so it returns the end position + 1 as the index. This makes retrieval of slicing source string much easier, as we won't be offset by one. If we want to get the text in the source string, we can use the slicing as: ```go exp, err := regexp.Compile(`�\d{5}(?:[-\s]\d{4})?�`) log_error(err) pincode_file, err := os.ReadFile("pincode.txt") log_error(err) match_index := exp.FindIndex(pincode_file) if len(match_index) > 0 { // Get the slice of the original string from start to end index sliced_string := pincode_file[match_index[0]:match_index[1]] log.Printf("%q ", sliced_string) } ``` ```console $ go run main.go "12345-1234" ``` So, we can access the string from the source text without calling other functions, simply slicing the original string will retrieve the expected results. This is the convention of the golang standard library to make the end index exclusive. Similarly, the `FindAllIndex` method can be used to get a list of list of such indexes of matched strings. ```go exp, err := regexp.Compile(`�\d{5}(?:[-\s]\d{4})?�`) log_error(err) pincode_file, err := os.ReadFile("pincode.txt") log_error(err) match_indexes := exp.FindAllIndex(pincode_file) log.Printf("%T ", match_indexes) log.Println(match_indexes) ``` ```console $ go run main.go [][]int [[9 19] [26 31]] ``` The above example gets the list of slices as for all the search pattern indexes in the source string/text. We can iterate over the list and get each matched string index. ## Find Submatch the `regexp` package also has a utility function for finding the sub-match of a given regular expression. The method returns a list of strings (or slices of bytes) containing the leftmost match and the sub-matches in that match. This also has the `All` version which instead of returning a single match i.e. the leftmost match it returns all the matches and the corresponding sub-matches. ```go package main import ( "log" "regexp" ) func main() { str := "abc@def.com is the mail address of 8th user with id 12" exp := regexp.MustCompile( `([a-zA-Z0-9]+@[a-zA-Z0-9]+\.[a-zA-Z]{2,})` + `|(email|mail)|` + `(\d{1,3})`)`, ) match := exp.FindStringSubmatch(str) log.Println(match) matches := exp.FindAllStringSubmatch(str, -1) log.Println(matches) } ``` ```console $ go run main.go [abc@def.com abc@def.com ] [[abc@def.com abc@def.com ] [mail mail ] [8 8] [12 12]] ``` The above example uses a regex to compute a few things like email address, words like `mail` or `email`, also a number of up to 3 digits. The `|` between these expressions indicates any combination of these three things would be matched for the regular expression. The [FindStringSubmatch](https://pkg.go.dev/regexp#Regexp.FindStringSubmatch) method, takes in a string as the source and returns a slice of the matching pattern. The first element is the leftmost match in the string source for the given regular expression, and the subsequent elements are the sub-matches in the matched string. We can now move a little step ahead for actually understanding the sub-match in a regular expression. ```go str := "abe21@def.com is the mail address of 8th user with id 124" exp := regexp.MustCompile( `([a-zA-Z]+(\d*)[a-zA-Z]*@[a-zA-Z]*(\d*)[a-zA-Z]+\.[a-z|A-Z]{2,})` + `|(mail|email)` + `|(\d{1,3})`, ) match := exp.FindStringSubmatch(str) log.Println(match) matches := exp.FindAllStringSubmatch(str, -1) log.Println(matches) ``` ```console $ go run main.go [abe21@def.com abe21@def.com 21 ] [[abe21@def.com abe21@def.com 21 ] [mail mail ] [8 8] [124 124]] ``` In the above example, there are a few things to take away, let us break it down into small pieces. We have a regex for matching either a mail address, the word `mail` or `email`, or a number up to 3 digits. The regex is a bit different from the previous example for understanding the sub-matches within an expression. We find the sub-matches in the string which will be handled by the `FindStringSubmatch`. This function takes in a string and returns a list of strings that are the leftmost matches in the source string. First, we need to understand the regex to get a clear idea of the code snippet. The first sub-match is for the email address. However, we use `[a-zA-Z]` in the picking of the username and the domain name, we don't want to directly match numbers yet. The goal of this regex is to pick up numbers inside an email address. So, we can have 1 or more characters `[a-zA-z]+`, followed by 0 or more digits `(\d*)`, and again we can have 0 or more characters `[a-zA-Z]*`. The `+` is for 1 or more, `*` is for 0 or more, `\d` is for digits. After this, we have the `@` as a compulsory character in the mail, followed by the same sequence in the username i.e. 1 or more characters, 0 or more digits, 0 or more characters. Finally, we have the `.com` and the domain name extensions, as a group of 2 or more characters `[a-z|A-Z]{2,}`. So, the regex accepts an email, with a sub-match of the number anywhere in the username or the domain name. The `FindStringSubmatch` function lists out the sub-matches for the leftmost(first) match of the regex in the source string. So it finds the string `abc21@def.com` which is the email id. This regex `([a-zA-Z]+(\d*)[a-zA-Z]*@[a-zA-Z]+(\d*)[a-zA-Z]*\.[a-z|A-Z]{2,})` has two sub-matches `\d*` in the username part and also in the domain part. So, the list returns the email address as the match and the match as itself, as well as the number in the sub-match inside the mail address. So the result, `[abc21@def.com abc21@def.com 21 ]`, there are a few empty string sub-matches because the second sub-match for the domain name number returns an empty string. Similarly, the `FindAllStringSubmatch` will return the list of all the matches in the source string. The other matches don't have any sub-matches in the regular expression so it just gets the match and the sub-match as itself in the case of string `mail`, digit `8`, and digit `124`. We can also use this example from a file as a slice of bytes. This will return a list of slice of slice of bytes instead of slice of slice of strings. ```go package main import ( "log" "regexp" ) func main() { exp := regexp.MustCompile( `([a-zA-Z]+(\d*)[a-zA-Z]*@[a-zA-Z]*(\d*)[a-zA-Z]+\.[a-z|A-Z]{2,})` + `|(mail|email)` + `|(\d{1,3})`, ) email_file, err := os.ReadFile("subtext.txt") log_error(err) mail_match := exp.FindSubmatch(email_file) log.Printf("%s ", mail_match) mail_matches := exp.FindAllSubmatch(email_file, -1) //log.Println(mail_matches) log.Printf("%s ", mail_matches) } ``` ```txt # submatch.txt abc21@def.com is the mail address of user id 1234 The email address abe2def.com is of user name abc a2be.@def.com Email address: abe@de2f.com, User id: 45 johndoe@example.com jane.doe123@example.com janedoe@example.co.uk john123@example.org janedoe456@example.net ``` ```console $ go run main.go [][]unit8 [abc21@def.com abc21@def.com 21 ] [][][]unit8 [ [abc21@def.com abc21@def.com 21 ] [mail mail ] [123 123] [4 4] [email email ] [2 2] [2 2] [mail mail ] [abe@de2f.com abe@de2f.com 2 ] [45 45] [johndoe@example.com johndoe@example.com ] [doe123@example.com doe123@example.com 123 ] [janedoe@example.co janedoe@example.co ] [john123@example.org john123@example.org 123 ] [janedoe456@example.net janedoe456@example.net 456 ] ] ``` As we can see from the dummy email ids and some random text, we are able to match the email ids and the numbers in them as the sub-match of the string. This is a `[][]unit8` in case of the `FindSubmatch` and `[][][]unit8` in case of `FindAllSubmatch`. The working remains the same for the bytes as it was for the strings. ### Find Submatch Index We also have the [FindSubmatchIndex](https://pkg.go.dev/regexp#Regexp.FindSubmatchIndex) and [FindAllSubmatchIndex](https://pkg.go.dev/regexp#Regexp.FindAllSubmatchIndex), and the string variants to get the index(es) of the sub-matches in the pattern picked from the regular expression. ```go str := "abe21@def.com is the mail address of 8th user with id 124" exp := regexp.MustCompile( `([a-zA-Z]+(\d*)[a-zA-Z]*@[a-zA-Z]*(\d*)[a-zA-Z]+\.[a-z|A-Z]{2,})` + `|(mail|email)` + `|(\d{1,3})`, ) match := exp.FindStringSubmatch(str) match_index := exp.FindStringSubmatchIndex(str) log.Println(match) log.Println(match_index) ``` ```console $ go run main.go [abe21@def.com abe21@def.com 21 ] [0 13 0 13 3 5 9 9 -1 -1 -1 -1] ``` So this returns a list of pairs, here consider the list as `[(0, 13) (0, 13), (3, 5), (9, 9), (-1, -1), (-1, -1)]`. Because the list represents the start and end indexes of the sub-matches in the source string. The match is the first pair, i.e. at the first character `0` -&gt; `a`, and ends at space i.e. `13` character off by one. Then we have the sub-match itself with the same indexes. Then the `3` and `5` indicating the number sub-match `21` in the `abc21@def.com` it starts at 3 and ends at 5, so it occupies 3 and 4 characters in the source string. Similarly, the domain level number doesn't have any number in the source string so it has returned the domain name end index as an empty string. We had used `(\d*)` which can return 0 or more occurrences of the digit, so it returned no match in the case of the domain level name, hence we get the `9` as an empty string at the end of the sub-match for it. The rest are for the `email` or `mail`, and the lone digit sub-matches in the regular expression which we don't have in the first match in the source string. ```go str := "abe21@def.com is the mail address of 8th user with id 124" exp := regexp.MustCompile( `([a-zA-Z]+(\d*)[a-zA-Z]*@[a-zA-Z]*(\d*)[a-zA-Z]+\.[a-z|A-Z]{2,})` + `|(mail|email)` + `|(\d{1,3})`, ) match := exp.FindAllStringSubmatch(str, -1) match_indexes := exp.FindAllStringSubmatchIndex(str, -1) log.Println(match_indexes) ``` ```console $ go run main.go [[abe21@def.com abe21@def.com 21 ] [mail mail ] [8 8] [124 124]] [ [0 13 0 13 3 5 9 9 -1 -1 -1 -1] [21 25 -1 -1 -1 -1 -1 -1 21 25 -1 -1] [37 38 -1 -1 -1 -1 -1 -1 -1 -1 37 38] [54 56 -1 -1 -1 -1 -1 -1 -1 -1 54 57] ] ``` Similarly, we have used the [FindAllStringSubmatchIndex](https://pkg.go.dev/regexp#Regexp.FindAllStringSubmatchIndex) for getting the list of a slice of indexes(int) for the sub-match of the expression in the source string. The first element is the same as the previous example, the next match in the source string is `mail` which comes at index 21 and ends at the 24th index which golang does 24+1 as a convention. Similarly, the number `8` is matched at index `37` and the number `124` at index `54`, rest of the sub-matches for these matches are not present so it turns up to be -1 for the rest of the sub-matches. So this can also be used for the byte/uint8 type of variants with [FindSubmatchIndex](https://pkg.go.dev/regexp#Regexp.FindSubmatchIndex) and [FindAllSubmatchIndex](https://pkg.go.dev/regexp#Regexp.FindAllSubmatchIndex). ## Replace Patterns The replace method is used for replacing the matched patterns. The [ReplaceAll](https://pkg.go.dev/regexp#Regexp.ReplaceAll) and [ReplaceAllLiteral](https://pkg.go.dev/regexp#Regexp.ReplaceAllLiteral) with some string and byte slice variations can help us in replacing the source text with a replacement string against a regular expression. Let's start with a simple example of strings. ```go package main import ( "log" "regexp" ) func main() { str := "Gophy gophers go in the golang grounds" exp := regexp.MustCompile(`(Go|go)`) replaced_str := exp.ReplaceAllString(str, "to") log.Println(replaced_str) } ``` The above code can replace the string with regex with the replacement string. The regex provided in the `exp` variable has a pattern matching for `Go` or `go`. So, the `ReplaceAllString` method takes in two arguments as strings and returns the string after replacing the character. ```console $ go run replace.go Original: Gophy gophers go in the golang grounds Replaced: tophy tophers to in the tolang grounds ``` So, this replaces all the characters with `Go` or `go` in the source string with `to`. There is a special value available in the replacement string which can expand the regular expression literals. Since the regular expression consists of multiple literals/quantifiers, we can use those to replace or keep the string in a particular expression's evaluation. ```go str = "Gophy gophers go in the golang grounds" exp2 := regexp.MustCompile(`(Go|go)|(phers)|(rounds)`) log.Println(exp2.ReplaceAllString(str, "hop")) log.Println(exp2.ReplaceAllString(str, "$1")) log.Println(exp2.ReplaceAllString(str, "$2")) log.Println(exp2.ReplaceAllString(str, "$3")) ``` ```console $ go run replace.go hopphy hophop hop in the hoplang ghop Gophy go go in the golang g phy phers in the lang g phy in the lang grounds ``` The above code uses the regular expression match string have been replaced with the literals. So, the regex `(Go|go)|(phers)|(rounds)` has three parts `Go|go`, `phers`, and `rounds`. Each of the literals is separated by an operator, indicating either the match or all matches should be considered. In the first statement, we replace the regex with `hop` as you can see all the matches are replaced with the replacement string. For instance, the word `gophers` is replaced by `hophop`, because `go` and `phers` is matched separately and replaced. In the second statement, we replace the source string with `$1` indicating the first literal in the regex i.e. `Go|go`. These statements will expand the `$1` and keep the match only where the literal matches and rest are removed. So, `Gophy` is matched with `Go|go` so it is kept as is in the replacement. However, for `grounds`, the literal match for `$1` i.e. `Go|go` does not match and hence is not kept and removed, so the resulting string becomes `g` rest is substituted with an empty string. In the third print statement, the source string is replaced with the second literal `$2` i.e. `phers`. So if any string matches the literal, only those are kept and the rest are substituted by an empty string. So, `Gophy` or `Go` doesn't match `phers` and hence is replaced, however, `gophers` match the `phers` part and is kept as it is, but the `go` part is substituted. Similarly, for the fourth print statement, the source string is replaced with the third literal i.e. `rounds`. So if only the third literal is kept as is, the rest matching strings from the regex are substituted with an empty string. So, `grounds` remain as it is because it matches the `rounds` in the replacement literal. In short, we replace the literal back after replacing the regex patterns in the source string. This can be used to fine-tune or access specific literals or expressions in the regular expressions. ```go str = "Gophy gophers go in the golang grounds" exp2 := regexp.MustCompile(`(Go|go)|(phers)|(rounds)`) log.Println(exp2.ReplaceAllString(str, "$1$2")) str = "Gophy gophers go in the golang cophers grounds" log.Println(exp2.ReplaceAllString(str, "$1$3")) ``` ```console $ go run replace.go Gophy gophers go in the golang g Gophy go go in the golang co grounds ``` We can even concatenate and make some minor string literal adjustments to the replacement string. As we have done in the example, where both the `$1$2` are used as the replacement string. The two literals combine to make a string with `Go|go` and `phers`. So, we can see the result, in the first statement, `Gophy gophers go in the golang g`, all the character that have `Go|go` or `phers` are kept as it is(substituted the same string), the `grounds` however does not match and hence are replaced with the empty string(because the capture group `$1$2` does not match `rounds`). Similarly, for the third example, the `$1$3` i.e. `Go|go` or `rounds` are matched with the source string. So, we the `phers` in `gophers` and `cophers` does not match the capture group `$1$3` and hence is replaced by an empty group(string). However, the `Gophy`, `golang`, and `grounds` match the capture group and are replaced by that match string (which is the same string). If we want to avoid expansion of the strings as we did in the previous example with `$1` and other parameters, we can use the [ReplaceAllLiteral](https://pkg.go.dev/regexp#Regexp.ReplaceAllLiteral) or [ReplaceAllLiteralString](https://pkg.go.dev/regexp#Regexp.ReplaceAllLiteralString) to parse the string as it is. ```go str := "Gophy gophers go in the golang cophers grounds" exp2 := regexp.MustCompile(`(Go|go)|(phers)|(rounds)`) log.Println(exp2.ReplaceAllLiteralString(str, "$1")) ``` ``` $ go run replace.go $1phy $1$1 $1 in the $1lang co$1 g$1 ``` As we can see the `$1` is not expanded and parsed as it is for replacing the pattern in the regular expression. The `Go` is replaced with `$1` to look like `$1phy`, and similarly for the rest of the patterns. That's it from the 26th part of the series, all the source code for the examples are linked in the GitHub on the [100 days of Golang](https://github.com/Mr-Destructive/100-days-of-golang/tree/main/scripts/regex) repository. ## Conclusion This article covered the fundamentals of using the `regexp` package for working with regular expressions in golang. We explored the methods and `Regexp` type in the package with various methods available through the type interface. By exploring the examples and simple snippets, various ways for pattern matching, finding, and replacing were walked over and found. So, hopefully, the article might have found useful to you, if you have any queries, questions, feedback, or mistakes in the article, you can let me know in the discussion or on my social handles. Happy Coding :) https://www.meetgor.com/posts/django-htmx-chat-app Django + HTMX Chat application Introduction In this article, we will be creating a Django project, which will be a chat-room kind of application. The user needs Sun, 05 Feb 2023 00:00:00 UTC # Django + HTMX Chat application ## Introduction In this article, we will be creating a Django project, which will be a chat-room kind of application. The user needs to authenticate to the app and then and there he/she can create or join rooms, every room will have some name and URL associated with it. So, the user simply needs to enter the name of the room, which will be unique. The user can then simply enter the messages in the chat room. This is a core chat application that uses web sockets. The unique thing about this app will be that, we don't have to write a javascript client. It will all be handled by some HTMX magic. The web socket in the backend will definitely handle using Django channels. Demo:  [GitHub Repository](https://github.com/Mr-Destructive/django-htmx-chat) ### Requirements: * Django * Django-channels * daphne * HTMX * SQLite or any relational database Also if we want to use the application on a large and production scale: * Redis * channels_redis The code for this chat app is provided in the [GitHub repository](https://github.com/Mr-Destructive/django-htmx-chat). ## Setup for Django project We will create a simple Django project to start with. The project can have two apps, one for auth and the other for the chat. You can customize the way you want your existing project accordingly. This project is just a demonstration of the use of the chat application with websockets and Django channels. I'll call the project `backchat`, you can call it whatever you want. We will create a virtual environment and install Django in that virtual environment ```bash virtualenv .venv For Linux/macOS: source .venv/bin/activate For Windows: .venv\scripts�ctivate pip install django django-admin startproject backchat cd backchat ``` This will set up a base Django project. We can now work on the actual implementation of the Django project. Firstly, we will start with authentication. ## Adding basic Authentication and Authorization ### Creating the accounts app We can separate the authentication of the user from the rest of the project, by creating a separate app called `user` or `accounts` app. ```bash python manage.py startapp accounts ``` ### Creating a base user model We'll start by inheriting the [AbstractUser](https://docs.djangoproject.com/en/4.1/topics/auth/customizing/#using-a-custom-user-model-when-starting-a-project) the model provided in the `django.contrib.auth.models` module. The model has base fields like `username` and `password` which are required fields, and `email`, `first_name`, `last_name`, etc. which are not mandatory. It is better to create a custom model by inheriting the `AbstractUser` because if in the longer run, we need to make custom fields in the user model, it becomes a breeze. ```python # accounts/models.py from djnago.contrib.auth.models import AbstractUser class User(AbstractUser): pass ``` This creates a basic custom user rather than using the Django built-in user. Next, we need to make sure, Django understands the default user is the one we defined in the `accounts` app and not the default `User`. So, we just need to add a field called `AUTH_USER_MODEL` in the `settings.py` file. The value of this field will be the app name or the module name and the model in that python module that we need our custom user model to be set as the default user model. ```python # backchat/settings.py INSALLED_APPS = [ ... ... "accounts", ] # Append to the end of file AUTH_USER_MODEL = 'accounts.User' ``` ### Initial migrations for the Django project Now, this will get picked up as the default user model while referring to any processing related to the user. We can move into migrating the changes for the basic Django project and the user model. ```bash python manage.py makemigrations python manage.py migrate ```  ### Creating register view Further, we can add the views like register and profile for the accounts app that can be used for the basic authentication. The Login and Logout views are provided in the `django.contrib.auth.views` module, we only have to write our own register view. I will be using function-based views to keep it simple to understand but it can simply be a class-based view as well. To define a view first, we need to have form, a user registration form. The form will inherit from the [UserCreationForm](https://docs.djangoproject.com/en/4.1/topics/auth/default/#django.contrib.auth.forms.UserCreationForm) form which will make the bulk of the task for us at the user registration. We can simply then override the Meta class with the fields that we want to display, so hence we just keep the `username` and the `password` fields. The form can be customized by adding in the widget attribute and specifying the classes used in them. ```python # accounts/forms.py from accounts.models import User from django.contrib.auth.forms import UserCreationForm class UserRegisterForm(UserCreationForm): class Meta: model= User fields = ['username', 'password1', 'password2'] ``` This will give us the `UserRegisterForm` form that can be used for displaying in the register view that we will create in the next step. We will have to create the register view that will render the form for user registration and will also process the form submission. ```python # accounts/views.py from django.contrib import messages from django.shortcuts import redirect, render from accounts.forms import UserRegisterForm def register(request): if request.method == "POST": form = UserRegisterForm(request.POST) if form.is_valid(): form.save() username = form.cleaned_data.get("username") messages.success( request, f"Your account has been created! You are now able to log in" ) return redirect("login") else: form = UserRegisterForm() return render(request, "accounts/register.html", {"form": form}) ``` The above register view has two cases, one for the user requesting the registration form and the second request when the user submits the form. So, when the user makes a get request, we load an empty form `UserRegisterForm` and render the `register` template with the form. We will make the templates later. So, the template is just rendered when the user wants to register and when the user submits the form(sends a post request) we get the details from the post request and make it an instance of `UserRegisterForm` and save the form if it is valid. We then redirect the user to the login view (will use the default one in the next section). We parse the message as the indication that the user was created. ### Adding URLs for Authentication and Authorization Once, we have the register view setup, we can also add login and logout views in the app. But, we don't have to write it ourselves, we can override them if needed, but we'll keep the default ones. We will use the [LoginView](https://docs.djangoproject.com/en/4.1/topics/auth/default/#django.contrib.auth.views.LoginView) and the [LogoutView](https://docs.djangoproject.com/en/4.1/topics/auth/default/#django.contrib.auth.views.LogoutView) view which are class-based views provided in the `django.contrib.auth.views` module. We will provide the respective templates for each of these views. ```python # accounts/urls.py from django.urls import path from django.contrib.auth import views as auth_views import user.views as user_views urlpatterns = [ path("register/", user_views.register, name="register"), path( "login/", auth_views.LoginView.as_view(template_name="accounts/login.html"), name="login", ), path( "logout/", auth_views.LogoutView.as_view(template_name="accounts/logout.html"), name="logout", ), ] ``` We have named the URLs as `register`, `login`, and `logout` so that we can use them while rendering the links for them in the templates. Now, we also need to include the URLs from the accounts app in the project URLs. We can do that by using the `include` method and specifying the app name with the module where the urlpatterns are located. ```python # backchat/urls.py from django.contrib import admin from django.urls import include, path urlpatterns = [ path("admin/", admin.site.urls), path("auth/", include("accounts.urls")), ] ``` So, we have routed the `/auth` path to include all the URLs in the accounts app. So, the login view will be at the URL `/auth/login/` , and so on. Also, we need to add the `LOGIN_REDIRECT_URL` and `LOGIN_URL` for specifying the url name which will be redirected once the user is logged in and the default login url name respectively. ```python # backchat / settings.py LOGIN_REDIRECT_URL = "index" LOGIN_URL = "login" ``` We are now almost done with the view and routing part of the accounts app and can move into the creation of templates. ### Adding Templates for authentication views We need a few templates that we have been referencing in the views and the URLs of the accounts app in the project. So there are a couple of ways to use templates in a Django project. I prefer to have a single templates folder in the root of the project and have subfolders as the app which will have the templates specific to those apps. I usually create a `base.html` file in the templates folder and use that for inheriting other templates. So, we will have to change one setting in the project to make sure it looks at `templates/` from the root of the project. ```bash # backchat/settings.py import os ... ... TEMPLATES = [ { "BACKEND": "django.template.backends.django.DjangoTemplates", "DIRS": [ os.path.join(BASE_DIR, "templates"), ], "APP_DIRS": True, "OPTIONS": { "context_processors": [ "django.template.context_processors.debug", "django.template.context_processors.request", "django.contrib.auth.context_processors.auth", "django.contrib.messages.context_processors.messages", ], }, }, ] ``` Then create the folder in the same path as you see your `manage.py` file. ```bash mkdir templates ```  #### Creating the base template The below will be the base template used for the chat application, you can custmize it as per your needs. ```html <!DOCTYPE html> <html> <head> <meta charset="utf-8" /> <title>Chat App</title> {% load static %} <script src="https://unpkg.com/htmx.org@1.8.5"></script> </head> <body> {% if user.is_authenticated %} <a href="#"logout' %}">Logout</a> {% else %} <a href="#"login' %}">Login</a> {% endif %} <h1>Back Chat</h1> {% block base %} {% endblock %} </body> </html> ``` We have included the [htmx](https://htmx.org/docs/#installing) package as the script into this template as we will be using it in the actual part of chat application. #### Creating the Register Template ```html # templates / accounts / register.html {% extends 'base.html' %} {% block base %} <div class="content-section"> <form method="POST"> {% csrf_token %} <fieldset class="form-group"> <legend class="border-bottom mb-4">Register Now</legend> {{ form.as_p }} </fieldset> <div class="form-group"> <button class="btn btn-outline-info" type="submit">Sign Up</button> </div> </form> <div class="border-top pt-3"> <small class="text-muted"> Already Have An Account? <a class="ml-2" href="#"login' %}">Log In</a> </small> </div> </div> {% endblock %} ```  #### Creating the Login Template ```html # templates / accounts / login.html {% extends 'base.html' %} {% block base %} <div class="content-section" id="login"> <form method="POST" > {% csrf_token %} <fieldset class="form-group"> <legend class="border-bottom mb-4">LOG IN</legend> {{ form.as_p }} </fieldset> <div class="form-group"> <button class="btn btn-outline-info" type="submit">Log In</button> </div> </form> <div class="border-top pt-3"> <small class="text-muted"> Register Here <a class="ml-2" href="#"register' %}">Sign Up</a> </small> </div> </div> {% endblock %} ```  Creating the Logout Template ```html # templates / accounts / logout.html {% extends 'base.html' %} {% block base %} <h2>You have been logged out</h2> <div class="border-top pt-3"> <small class="text-muted"> <a href="#"login' %}">Log In Again</a> </small> </div> {% endblock %} ``` ## Install and setup channels We will be using channels to create long-running connections, it is a wrapper around Django's asynchronous components and allows us to incorporate other protocols like web sockets and other asynchronous protocols. So, we will be using the Django channels package that will allow us to use the WebSocket protocol in the chat application. [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) is a communication protocol(set of rules and standards to be followed) that allows both the client and server to send and receive messages(communication). To install Django channels, we can use pip and install channels with daphne which will be used as the web server gateway interface for asynchronous web applications. ```bash pip install -U channels["daphne"] ``` So this will install the latest version of channels and daphne. We now have [channels](https://channels.readthedocs.io/en/stable/installation.html) in our Django project, we simply need to configure the `CHANNEL_LAYER` config for specifying the backend used that can be `Redis`, `In-Memory`, or others. We need to add `channels` , `daphne` to the `INSALLED_APPS` config of the project. Make sure the `daphne` app is on top of the applications list. ```bash # backchat/settings.py ... ... INSALLED_APPS = [ "daphne", ... ... "channels", ] ASGI_APPLICATION = "backchat.asgi.application" ... ... # For InMemory channels CHANNEL_LAYERS = { 'default': { "BACKEND": "channels.layers.InMemoryChannelLayer", } } # For Redis CHANNEL_LAYERS = { "default": { "BACKEND": "asgi_redis.RedisChannelLayer", "CONFIG": { "hosts": [("redis-host-url", 6379)], }, }, } ``` ### Redis Configuration (Optional) You can either use the [InMemoryChannelLayer](https://channels.readthedocs.io/en/latest/topics/channel_layers.html) or you can use them `RedisChannelLayer` for the backend of the chat app. There are other types of backends like `Amazon SQS` services, `RabbitMQ`, `Kafka`, `Google Cloud Pub/Sub`, etc. I will be creating the app with only the `InMemoryChannelLayer` but will provide a guide for redis as well, both are quite similar and only have a few nuances. We need to install [channels_redis](https://github.com/django/channels_redis/) it for integrating redis in the Django project with channels. ```bash pip install channels_redis ``` So, this will make the `channels_redis` package available in the project, we use this package for real-time storage, in the case of the chat app, we might use it for storing messages or room details, etc. ## Creating the Chat App Further, we now can create another app for handling the rooms and chat application logic. This app will have its own models, views, and URLs. Also, we will define consumers and routers just like views and URLs but for asynchronous connections. More on that soon. So, let's create the `chat` app. ```bash python manage.py startapp chat ``` Then we will add the chat app to the `INSALLED_APPS` config. ```python # backchat/settings.py INSALLED_APPS = [ ... ..., "chat", ] ``` So, this will make sure to load the chat app in the project. Whenever we run any commands like migrations or running the server, it keeps the app in the `INSALLED_APPS` checked up. ### Defining models This is optional, but we'll do it for since we are making a Django app. We already have an auth system configured, adding rooms and authorizing the users will become easier then. So, let's create the models for the chat app which will be the `Room`. ```python # chat/models.py from django.db import models from accounts.models import User class Room(models.Model): name = models.CharField(max_length=128) slug = models.SlugField(unique=True) users = models.ManyToManyField(User) def __str__(self): return self.name class Message(models.Model): room = models.ForeignKey(Room, on_delete=models.CASCADE) user = models.ForeignKey(User, on_delete=models.CASCADE) message = models.TextField() created_at = models.DateTimeField(auto_now_add=True) def __str__(self): return ( self.room.name + " - " + str(self.user.username) + " : " + str(self.message) ) ``` So, we simply have the name which will be taken from the user, and the slug which will be served as the URL to the room. The users are set as [ManyToManyField](https://docs.djangoproject.com/en/4.1/ref/models/fields/#django.db.models.ManyToManyField) since one room can have multiple users and a user can be in multiple rooms. Also we are creating the model `Message` that will be storing the room and the user as the foreign key and the actual text as the message, we can improve the security by encrypting the text but it's not the point of this article. We have set the `created_at` the field which will set the time when the object was created. Finally, the dunder string method is used for representing the message object as a price of the concatenation of strings of room name, username, and the message. This is useful for admin stuff as it makes it easier to read the object rather than the default id. Now, once the models are designed we can migrate the models into the database. ``` python manage.py makemigrations python manage.py migrate ```  We now have a decent model structure for the chat application. We can now start the crux of the application i.e. consumers and routing with channels. ### Writing consumers and routers for WebSockets So, we start with the bare bones provided in the tutorial on the channel [documentation](https://channels.readthedocs.io/en/stable/tutorial/part_3.html). We create a class (consumer) called `ChatConsumer` which inherits from the `AsyncWebsocketConsumer` provided by the `channels.generic.websocket` module. This has a few methods like `connect`, `disconnect`, and `receive`. These are the building blocks of a consumer. We define these methods as they will be used for communication via the WebSocket protocol through the channels interface. In the following block of code, we are essentially doing the following: * Accepting connection on the requested room name * Sending and Receiving messages on the room/group * Closing the WebSocket connection and removing the client from the room/group ```python # chat/consumers.py import json from asgiref.sync import sync_to_async from channels.generic.websocket import AsyncWebsocketConsumer from chat.models import Room, Message class ChatConsumer(AsyncWebsocketConsumer): async def connect(self): self.room_name = self.scope["url_route"]["kwargs"]["room_slug"] self.room_group_name = f"chat_{self.room_name}" self.user = self.scope["user"] await self.channel_layer.group_add( self.room_group_name, self.channel_name ) await self.accept() async def disconnect(self, close_code): await self.channel_layer.group_discard( self.room_group_name, self.channel_name ) async def receive(self, text_data): text_data_json = json.loads(text_data) message = text_data_json["message"] username = self.user.username await self.channel_layer.group_send( self.room_group_name, { "type": "chat_message", "message": message, "username": username, } ) async def chat_message(self, event): message = event["message"] username = event["username"] await self.send( text_data=json.dumps( { "message": message, "username": username } ) ) ``` #### Accept the WebSocket connection Here, room and group more or less mean the same thing but are different in different contexts. For instance, the group refers to the collection of clients which are connected to a channel(communication between consumer and web socket) and the Room is referring to the collection of clients connected to a single connection stream like a literal room. So we can say, the group is a technical term and the room is more of a layman's term for the same thing. The method `connect` is called when the client establishes a websocket connection. When that happens, the function gets the room slug from the URL of the client and stores `room_name` which is a string. It creates a separate variable called `room_group_name` by prepending the `chat_` string to the `room_name`, it also gets the currently logged-in user from the request. It then adds the `channel_name` to the group named `room_group_name`. The `channel_name` is a unique identifier to the connection/consumer in the channel. By adding the `channel_name`, the consumer then can broadcast the message to all the channels within the group. Finally, the function accepts the connection, and a **webcoket connection is established from both ends, connection is sent from the client and is now accepted from the backend.** #### Disconnect from the WebSocket connection When the client sends a close connection request to the server, the `disconnect` method is triggered and it basically removes the `channel_name` from the group i.e. the group name `room_group_name` whatever the string it happens to be. So, it basically removes the client from the broadcast group and hence it can't receive or send messages through the websocket since it has been closed from both ends. If you would have paid attention to the `close_code` in-method parameter, it is not being used currently. However, we can use it for checking why the connection was closed, as the `close_code` is a numeric value just like the status code in the web request for letting the server know the reason for disconnecting from the client. #### Receive a message from the WebSocket connection The `recieve` method is the core of the consumer as it is responsible for all the logic and parsing of the message and broadcasting of the messages from the clients to the group channels. The function takes in a parameter called `text_data` and it is sent from the client through websocket, so it is JSON content. We need to get the actual message from the JSON object or any other piece of content from the client. So, we deserialize (convert the JSON object to python objects) the received payload, and get the value from the key `message`. The key is the input name or id from the client sending the request through the web socket, so it can be different depending on the frontend template(we'll see the front end soon as well). We get the user from the scope of the consumer as we previously initialized it in the connect method. This can be used for understanding which user has sent the message, it will be used later on as we send/broadcast the message to the group. Now, the final piece in the receive method is the `channel_layer.group_send` method, this method as the name suggests is used to send or broadcast the received message to the entire group. The method has two parameters: 1. The name of the group 2. The JSON body containing the message and other details The method is not directly sending the message but it has a type key in the JSON body which will be the function name to call. The function will simply call the other funciton mentioned in the type key in the dict. The following keys in the dict will be the parameters of that funciton. In this case, the funciton specified in the `type` key is `chat_message` which takes in the `event` as the parameter. This event will have all the parameters from the `group_send` method. So, the `chat_message` will load in this message, username, and the room name and then call the `send` method which actually sends the message as a JSON payload to the WebSocket connection which will be received by all the clients in the same group as provided in the `room_group_name` string. ### Adding Routers for WebSocket connections So, till this point have consumers, which are just like views in terms of channels. Now, we need some URL routes to map these consumers to a path. So, we will create a file/module called `routing.py` which will look quite similar to the `urls.py` file. It will have a list called `websocket_urlpatterns` just like `urlpatterns` with the list of `path`. These paths however are not `http` routes but will serve for the `WebSocket` path. ```python # chat / routing.py from django.urls import path from chat import consumers websocket_urlpatterns = [ path('chat/<str:room_slug>/', consumers.ChatConsumer.as_asgi()), ] ``` In the above code block, we have defined a URL for the web socket with the path as `/chat/<room_slug>` where room_name will be the `slug` for the room. The path is bound with the consumer-defined in the `consumers.py` module as `ChatConsumer`. The `as_asgi` method is used for converting a view into an ASGI-compatible view for the WebSocket interface. ### Setting up ASGI Application We are using the ASGI application config rather than a typical WSGI application which only works one request at a time. We want the chat application to be asynchronous because multiple clients might send and receive the messages at the same time, we don't want to make a client wait before the server process a message from another client, that's just the reason we are using WebSocket protocol. So, we need to also make sure, it makes the http request and also add our websocket config from the chat app we created in the previous sections. So, inside the `asgi.py` file in the project config module, we need to make some changes to include the chat application configurations. ```python # backchat / asgi.py import os from django.core.asgi import get_asgi_application from channels.auth import AuthMiddlewareStack from channels.routing import ProtocolTypeRouter, URLRouter os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'backchat.settings') from chat import routing application = ProtocolTypeRouter({ 'http': get_asgi_application(), "websocket":AuthMiddlewareStack( URLRouter( routing.websocket_urlpatterns ) ) }) ``` We will override the `application` config which is a component used for routing different types of protocols for the `ASGI` application. We have set the two keys, `http` and `websocket` in our application. The `http` type of requests will be served with the `get_asgi_application` application which is used for running the application in an ASGI environment. For `websocket` type of requests, we are setting the [AuthMiddlewareStack](https://channels.readthedocs.io/en/latest/topics/authentication.html) which helps in authenticating the users requesting the WebSocket connection and allows only authorized users to make a connection to the application. The [URLRouter](https://channels.readthedocs.io/en/stable/topics/routing.html) is used for mapping the list of URL patterns with the incoming request. So, this basically serves the request URL with the appropriate consumer in the application. We are parsing in the `websocket_urlpatterns` as the list of URLs that will be used for the WebSocket connections. Now, we run the server, we should be seeing the `ASGI` server serving our application rather than the plain WSGI application. ``` $ python manage.py runserver Watching for file changes with StatReloader Performing system checks... System check identified no issues (0 silenced). February 05, 2023 - 09:22:45 Django version 4.1.5, using settings 'backchat.settings' Starting ASGI/Daphne version 4.0.0 development server at http://127.0.0.1:8000/ Quit the server with CONTROL-C. ``` The application is not complete yet, it might not have most components working functional yet. So, we'll now get into making the user interfaces for the application, to create, join, and view rooms in the application. ### Adding Views for Chat Rooms We will have a couple of views like create room page, the join room page, and the chat room page. We will define each view as a distinct view and all of them will require authenticated users. ```python # chat / views.py import string import random from django.contrib.auth.decorators import login_required from django.shortcuts import render, reverse, redirect from django.utils.text import slugify from chat.models import Room @login_required def index(request, slug): room = Room.objects.get(slug=slug) return render(request, 'chat/room.html', {'name': room.name, 'slug': room.slug}) @login_required def room_create(request): if request.method == "POST": room_name = request.POST["room_name"] uid = str(''.join(random.choices(string.ascii_letters + string.digits, k=4))) room_slug = slugify(room_name + "_" + uid) room = Room.objects.create(name=room_name, slug=room_slug) return redirect(reverse('chat', kwargs={'slug': room.slug})) else: return render(request, 'chat/create.html') @login_required def room_join(request): if request.method == "POST": room_slug = request.POST["room_slug"] room = Room.objects.get(slug=room_slug) return redirect(reverse('chat', kwargs={'slug': room.slug})) else: return render(request, 'chat/join.html') ``` In the above views module, we have added 3 views namely `index` as the room page, `room_create` for the room creation page, and the `room_join` for the room join page. The index view is a simple get request to the provided slug of the room, it gets the slug from the URL from the request and fetches an object of the room associated with that slug. Then it renders the room template with the context variables like the name of the room and the slug associated with that room. The `room_create` view is a simple two-case view that either can render the room creation page or process the submitted form and create the room. Just like we used in the `register` view in the accounts app. When the user will send a `GET` request to the URL which we will map to `/create/` shortly after this, the user will be given a form. So, we will render the `create.html` template. We will create the html template shortly. If the user has sent a `POST` request to the view via the `/create` URL, we will fetch the name field in the sent request and create a unique identifier with the name of the room. We will slugify the concatenation of the name with the uid and will set it as the slug of the room. We will then simply create the room and redirect the user to the `room` page. The `room_join` view also is a two-case view, where the user can either request the join room form or send a slug with the form submission. If the user is requesting a form, we will render the `join.html` template. If the user is submitting the form, we will fetch the room based on the slug provided and redirect the user to the `room` page. So, the `room_join` and `room_create` views are quite similar, we are fetching an already existing room in the case of the join view and creating a new instance of room in the create view. Now, we will connect the views to the URLs and finally get to the templates. ### Connecting Views and URLs We have three views to route to a URL. But we will also have one additional URL which will be the home page for the application, on that page the user can choose to either join or create a room. We have the room creation, join the room and the room view to be mapped in this URL routing of the app. ```python # chat / urls.py from django.urls import path from django.views.generic import TemplateView from chat import views urlpatterns = [ path("", TemplateView.as_view(template_name="base.html"), name='index'), path("room/<str:slug>/", views.index, name='chat'), path("create/", views.room_create, name='room-create'), path("join/", views.room_join, name='room-join'), ] ``` So, the first route is the home page view called `index`, we have used the [TemplateView](https://docs.djangoproject.com/en/4.1/ref/class-based-views/base/#templateview) which will simply render the template provided. We don't have to create a separate view for just rendering a template. We already have defined the `base.html` template while setting up the `accounts` app. This will be the same template, we will add some more content to the template later on. The URL mapped is the `/` since we will add the URLs of this app to the project URLs with an empty `""` path. The second route is used for the room index page, i.e. where the user will be able to send and receive messages. The path is assigned as `/room/<str:slug>/` indicating a parameter called slug of type string will be used in accessing a particular room. The URL will be bound to the index view that we created in the views file, which fetches the room based on the slug, so here is where the slug will be coming from. The name of the URL-View route will be `chat` but you can keep it as per your requirements. The URL name is really handy for use in the templates. The third route is for the room creation page. The `/create/` URL will be bound to the `room_create` view, as we discussed, it will serve two purposes, one to render the form for creating the room, and the other for sending a `POST` request to the same path for the creating the Room with the name provided. The name is not required but helps in identifying and making it user-friendly. The name of this URL is set as `room-create`. The final route is for joining the room, the `/join/` URL will be triggering the `room_join` view. Similar to the `room-create` URL, the URL will render the join room form on a `GET` request, fetch the room with the provided slug and redirect to the room page. Here, the slug field in the form will be required for actually finding the matching room. The name of the URL route is set as `room-join`. We now add the URLs from the chat app to the project URLs. This will make the `/` as the entry point for the chat application URLs. ```python # backchat / urls.py from django.contrib import admin from django.urls import path, include urlpatterns = [ path("admin/", admin.site.urls), path("auth/", include('accounts.urls')), path("", include('chat.urls')), ] ``` Hence the process is completed for the backend to process the message, it then is dependent on the client to process and render the message. Till HTMX was not a thing! We won't have to write a single line of javascript to receive and handle the WebSocket connection! ### Creating Templates and adding htmx We now move into the actual frontend or creating the template for actually working with the rooms and user interaction. We will have three pieces of templates, a room creates the page, a room join page, and a room chat page. As these template names suggest, they will be used for creating a room with the name, joining the room with the slug, and the room chat page where the user will send and receive messages. Let/s modify the base template first. ```html <!DOCTYPE html> <html> <head> <meta charset="utf-8" /> <title>Chat App</title> {% load static %} <script src="https://unpkg.com/htmx.org@1.8.5"></script> </head> <a href="#"index' %}">Home</a> {% if user.is_authenticated %} <a href="#"logout' %}">Logout</a> {% else %} <a href="#"login' %}">Login</a> {% endif %} <body hx-ext="ws"> <h1>Back Chat</h1> {% block base %} <a href="#"room-join' %}">Join Room</a> <a href="#"room-create' %}">Create Room</a> {% endblock %} </body> </html> ```  #### Create Room Template We will have to render the form with a field like `name` for setting it as the name of the room, it is not required but again, it makes it easier for the user to find the name of the room a bit more friendly than random characters. ```html # templates / chat / create.html {% extends 'base.html' %} {% block base %} <form method='post' action='{% url 'room-create' %}'> {% csrf_token %} <input name='room_name' id='room_name' placeholder='Room Name'> <input type='submit' id="submit"> </form> {% endblock %} ```  The template is inherited from the `base.html` template and we render a form with the `room_name` input. The form will be submitted to the URL named `room-create` hence it will send a `POST` request to the server where it will create the room and further process the request. #### Join Room Template The join room template is similar to the create room template except it gets the slug of the room from the user rather than the name which is not a unique one to join the room. ```html # templates / chat / join.html {% extends 'base.html' %} {% block base %} <form method='post' action='{% url 'room-join' %}'> {% csrf_token %} <input name='room_slug' id='room_slug' required='true' placeholder='Room Code'> <input type='submit' id="submit"> </form> {% endblock %} ```  The form will be submitted to the URL named `room-join` hence it will send a `POST` request to the server where it will fetch the room and further process the request. ### Room Template (HTMX code) Now, time for the actual ingredient in the application, some HTMX magic! This template, as the two templates above inherit from the base template, that's the same thing. But it has a special `div` with the attribute [hx-ws](https://htmx.org/attributes/hx-ws/) which is used for using attributes related to the web socket in the htmx library. The `connect` value is used for connecting to a WebSocket. The value of the attribute must be set to the URL of the WebSocket to be connected. In our case, it is the URL path from the `routing` app as `/chat/<room_slug>/`. This simply will connect the client to the WebSocket from the backend. The other important attribute is the `send` which is used for sending a message to the connected web socket. ```html # templates / chat / room.html {% extends 'base.html' %} {% block base %} <h2>{{ name }}</h2> <div hx-ws="connect:/chat/{{ slug }}/"> <form hx-ws="send:submit"> <input name="message"> <input type="submit"> </form> </div> <div id='messages'></div> {% endblock %} ```  NOTE: The template has a div with the id `messages` which will be very important for sending the messages from the WebSocket to the client, so more on that when we use the HTMX part. For testing this template, you can create a room, and that will redirect you to the room template as we have seen in the views for the room creation. If you see something like `WebSocket CONNECT` it means, that the application has been able to establish a WebSocket connection to the backend, and we can be ready to accept messages and other stuff. ``` HTTP GET /chat/room/def_teas/ 200 [0.03, 127.0.0.1:38660] WebSocket HANDSHAKING /chat/def_teas/ [127.0.0.1:38666] WebSocket CONNECT /chat/def_teas/ [127.0.0.1:38666] ```  Till this point, we should have a running and almost complete application, though we just have a minor part missing that will be the most important part. ### Sending HTML response from backend for htmx We will be sending a fragment of HTML from the backend when the user sends a message, to broadcast it to the group. Let's make some changes to the application, especially to the receive method in the `ChatConsumer` of the chat application. ```python # chat / consumers.py ... ... async def receive(self, text_data): text_data_json = json.loads(text_data) message = text_data_json["message"] user = self.user username = user.username await self.channel_layer.group_send( self.room_group_name, { "type": "chat_message", "message": message, "username": username, } ) async def chat_message(self, event): message = event["message"] username = event["username"] # This is the crucial part of the application message_html = f"<div hx-swap-oob='beforeend:#messages'><p><b>{username}</b>: {message}</p></div>" await self.send( text_data=json.dumps( { "message": message_html, "username": username } ) ) ```    In the above snippet, we are just changing the final message object to include some HTML just simple. The HTML however has home htmx attributes like [hx-swap-oob](https://htmx.org/attributes/hx-swap-oob/) which will just update the specified DOM element to the content in the div. In this case, the DOM element is `#message` which is the id message present in the room template. We basically add the username and the message into the same id by appending it to the element. That's it, it would work and it would start showing the messages from the connected clients and broadcast them as well. There are some things to keep in mind while using htmx in the long run especially when the htmx 2.0 is released, it will have `ws` as a separate extension. It will have a bit of a different syntax than above. I have tried the latest version but doesn't seem to work. I'll just leave a few snippets for your understanding of the problem. ```html # templates / chat / room.html {% extends 'base.html' %} {% block base %} <h2>{{ name }}</h2> <div hx-ext="ws" ws-connect="/chat/{{ slug }}/"> <form ws-send> <input name="message"> </form> </div> <div id='messages'></div> {% endblock %} ``` I have added, the `hx-ext` as `ws` which is a htmx [extension for websocket](https://htmx.org/extensions/web-sockets/). This extension has websocket-specific attributes like `ws-connect` and `ws-send`. I have tried a combination like changing the htmx versions, adding submit value as the `ws-send` attribute, etc, but no results yet. I have opened a [discussion](https://github.com/bigskysoftware/htmx/discussions/1231) on GitHub for this issue, you can express your solution or views there. ### Adding some utility features for the chat app We can save messages, add and remove the users from the room according to the connection, and other stuff that can make this a fully-fledged app. So, I have made a few changes to the chat consumers for saving the messages and also updating the room with the users in the room. ```python # chat / consumers.py import json from asgiref.sync import sync_to_async from channels.generic.websocket import AsyncWebsocketConsumer from chat.models import Room, Message class ChatConsumer(AsyncWebsocketConsumer): async def connect(self): self.room_name = self.scope["url_route"]["kwargs"]["room_slug"] self.room_group_name = "chat_%s" % self.room_name self.user = self.scope["user"] await self.channel_layer.group_add( self.room_group_name, self.channel_name ) # Add the user when the client connects await self.add_user(self.room_name, self.user) await self.accept() async def disconnect(self, close_code): # Remove the user when the client disconnects await self.remove_user(self.room_name, self.user) await self.channel_layer.group_discard( self.room_group_name, self.channel_name ) async def receive(self, text_data): text_data_json = json.loads(text_data) message = text_data_json["message"] user = self.user username = user.username room = self.room_name # Save the message on recieving await self.save_message(room, user, message) await self.channel_layer.group_send( self.room_group_name, { "type": "chat_message", "message": message, "username": username, } ) async def chat_message(self, event): message = event["message"] username = event["username"] message_html = f"<div hx-swap-oob='beforeend:#messages'><p><b>{username}</b>: {message}</p></div>" await self.send( text_data=json.dumps( { "message": message_html, "username": username } ) ) @sync_to_async def save_message(self, room, user, message): room = Room.objects.get(slug=room) Message.objects.create(room=room, user=user, message=message) @sync_to_async def add_user(self, room, user): room = Room.objects.get(slug=room) if user not in room.users.all(): room.users.add(user) room.save() @sync_to_async def remove_user(self, room, user): room = Room.objects.get(slug=room) if user in room.users.all(): room.users.remove(user) room.save() ``` So, we have created a few methods like `save_message`, `add_user`, and `remove_user` which all are `synchronous` methods but we are using an asynchronous web server, so we add in the `sync_to_async` decorator which wraps a synchronous method to an asynchronous method. Inside the methods, we simply perform the database operations like creating a message object, and adding or removing the user from the room. That's only a few features that I have added, you can add to this application and customize them as per your needs. The code for this chat app is provided in the [GitHub repository](https://github.com/Mr-Destructive/django-htmx-chat). ## Conclusion So, from this post, we were able to create a simple chat app (frontendless) with Django and htmx. We used Django channels and HTMX to make a chat application without the need to write javascript for the client-side connection. Hope you found this tutorial helpful, do give your feedback and thoughts on it, I'll be eager to improve this post. Thank you for your patient listening. Happy Coding :) https://www.meetgor.com/posts/golang-command-line-args Introduction In the 25th post of the series, we will be taking a look into parsing of command line arguments in golang. We will be exploring how to do the basic Sat, 07 Jan 2023 00:00:00 UTC ## Introduction In the 25th post of the series, we will be taking a look into parsing of command line arguments in golang. We will be exploring how to do the basics of parsing and using the positional parameters or arguments from the command line in the program. By using standard library packages like `os` and `flag`, we can make powerful yet easy-to-build CLI apps and programs. ## Parsing Arguments from the command line (os package) We can use the os package to get the arguments from the command line in a go script. We have to use the Args variable in the os package. The `Args` variable is a slice of strings which thereby is the parsed arguments from the command line. * The first (0 index) Argument is the path to the program * The 1st index onwards are the actual arguments passed. ```go package main import ( "fmt" "os" ) func main() { args := os.Args fmt.Printf("Type of Args = %T ", args) fmt.Println(args[0], args[1]) } ``` ```bash $ go run main.go hello Type of Args = []string /tmp/go-build1414795487/b001/exe/main hello ``` In the above example, we can see that the `Args` is a slice of string and we can get the indices as per the arguments passed from the command line. If you don't parse any arguments and access the 1st argument as `os.Args[1]` it will result in an `index out of range` error. So, you need to first check if the argument is parsed and set a default value otherwise. ```go package main import ( "fmt" "os" "strconv" ) func main() { var port int var err error if len(os.Args) > 1 { port, err = strconv.Atoi(args[1]) if err != nil { panic(err) } } else { port = 8000 } fmt.Println(port) } ``` ```bash $ go run main.go 8000 $ go run main.go 7000 7090 $ go run main.go h panic: strconv.Atoi: parsing "h": invalid syntax ``` In the above example, we have declared the port variable as an integer and tried to see if we had an argument parsed from the command line using the len function and if there was a variable, we will simply cast it into an integer using the [strconv.Atoi](https://pkg.go.dev/strconv#Atoi) function. If there are any errors in the process, we log an error message and panic out of the program. So, this is how we can set default values or check for any arguments from the command line in golang. ### Get the number of args We can use the len function with the `Args` slice to get the total number of arguments from the command line. To ignore the first argument which would be the path to the program, we simply can slice the first element as `os.Args[1:]`. This will slice the list of the arguments from the first index till the last element in the slice. ```go package main import ( "fmt" "os" ) func main() { total_args := len(os.Args[1:]) fmt.Println("Total Args =", total_args) } ``` ```bash $ go run main.go hello world 56 Total Args = 3 ``` This will simply give us the number of arguments passed from the command line, excluding the first(0th) argument which is the default argument as the execution path of the current program. ### Iterate over all arguments We can use the simple for loop with range over the `os.Args` or `os.Args[1:]` for iterating over each of the arguments passed from the command line. ```go package main import ( "fmt" "os" ) func main() { for n, args := range os.Args { fmt.Println("Arg", n, "->", args) } /* // For excluding the 0th argument for n, args := range os.Args[1:] { fmt.Println("Arg", n, "->", args) } */ } ``` ```bash $ go run main.go hello world 56 Arg 0 -> /tmp/go-build2248205073/b001/exe/main Arg 1 -> hello Arg 2 -> world Arg 3 -> 56 ``` We can now iterate over the arguments passed from the command line using a simple for loop. We can further process these arguments per the program's requirements and need. ## Using flags package Golang has a package in its standard library called [flags](https://pkg.go.dev/flag) which allows us to parse flags and arguments from the command line with a lot of built-in features. For instance, a default value is easily parsed with a simple function parameter, help text in case of an error in parsing the arguments or flag, customization and freedom for choosing a data type for the type of argument, and so on. For a bare-bones and quick CLI program, the flag package is a great choice. ### Parse Typed Flags We can use typed flag values using the functions provided in the `flags` package like [IntVar](https://pkg.go.dev/flag#IntVar) for an integer value, [StringVar](https://pkg.go.dev/flag#StringVar) for string, [BoolVar](https://pkg.go.dev/flag#BoolVar) for boolean values and so on. Each function takes in 4 parameters and they set the value of the parsed variable from the parsed argument/flag from the command line. * The first parameter is a reference to the variable to store the value. * The second parameter is the name of the argument/flag to be read from the command line. * The third parameter is the default value of the variable. * The fourth parameter is the help text for that argument/flag. So, let's take the previous example of port number parsing from the command line. We can use the `flag.IntVar(&port, "p", 8000, "Provide a port number")`, this will set the value of the variable port from the command line as the value of `-p 6789` or the default value as `8000`. The help text will be used if the user has provided a non-integer or an invalid value as an error message. ```go package main import ( "flag" "fmt" ) func main() { var port int var dir string var publish bool flag.IntVar(&port, "p", 8000, "Provide a port number") flag.StringVar(&dir, "dir", "output_dir", "Directory") flag.BoolVar(&publish, "publish", false, "Publish the article") flag.Parse() fmt.Println(port) fmt.Println(dir) fmt.Println(publish) if publish { fmt.Println("Publishing article...") } else { fmt.Println("Article saved as Draft!") } } ``` ```bash $ go run flag.go 8000 output_dir false Article saved as Draft! $ go run flag.go -p 1234 1234 output_dir false Article saved as Draft! $ go run flag.go -p 1234 -dir site_out 1234 site_out false Article saved as Draft! $ go run flag.go -publish 8000 output_dir true Publishing article... ``` So, in the above, example, we have used a few types of values like `IntegerVar` for `port`, `StringVar` for `dir`, and `BoolVar` for `publish`. As explained earlier, the functions take 4 parameters in the same format, the reference to the variable to hold the parsed value, the name of the argument/flag, the default value the variable will hold, and the help text or usage string. The [BoolVar](https://pkg.go.dev/flag#BoolVar) is slightly different but it works logically well, if we parse `-publish` the value will be set as `true` and `false` otherwise. You can manually add the value like `-publish true` and so on but it is not mandatory and understood as true. In the above example, we have parsed different arguments in the output and displayed the values of these flags. If we don't specify a value, we can see the default value being parsed, in the case of the `bool` variable, the default value is taken as `false`. Hence we can see how easily we can use and parse flags from the command line in golang, it's simple, quick, and also extensible. For other data types, the flag package has functions like [Float64Var](https://pkg.go.dev/flag#Float64Var) for float64 values, [DurationVar](https://pkg.go.dev/flag#DurationVar) for time duration values and [TextVar](https://pkg.go.dev/flag#TextVar) for other types as inferred by the unmarshalling of the text. ### Set flags from the script We can set the value of a flag/argument from the script rather than from the command line using the [Set](https://pkg.go.dev/flag#Set) method in the flag package. The `Set` method takes in two values as parameters the name of the argument and the value of that argument to set as. It returns an error if any arise during the setting of the argument. ```go package main import ( "flag" "fmt" ) func main() { var port int var dir string var publish bool flag.IntVar(&port, "p", 8000, "Provide a port number") flag.StringVar(&dir, "dir", "output_dir", "Directory") flag.Parse() fmt.Println(port) fmt.Println(dir) flag.Set("dir", "dumps") fmt.Println(dir) } ``` ```bash $ go run flag.go -p 8080 8080 output_dir dumps ``` So, it is clearly visible that the value of an argument can be changed within the script, it also changes the value of the associated variable. Remember, we gave the two-parameter as strings so the first parameter is the name of the argument and not necessarily the variable name. ### Use Reference to arguments (pointers) Also, there are functions like [Int](https://pkg.go.dev/flag#Int), [Float64](https://pkg.go.dev/flag#Float64), [String](https://pkg.go.dev/flag#String), [Bool](https://pkg.go.dev/flag#Bool) in the flag package that can allow getting the values of the arguments without using the `Parse` method. We use the reference of the value stored in as the arguments instead of defining the variables as a data value; we have a pointer to that value of data. ```go package main import ( "flag" "fmt" ) func main() { port := flag.Int("p", 8000, "Provide a port number") dir := flag.String("dir", "output_dir", "Directory") publish := flag.Bool("publish", false, "Publish the article") help := flag.Bool("help", false, "Help") if *help { flag.PrintDefaults() } else { fmt.Println(*port) fmt.Println(*dir) flag.Set("dir", "dumps") fmt.Println(*dir) fmt.Println(flag.NFlag()) fmt.Println(flag.NArg()) fmt.Println(*publish) if *publish { fmt.Println("Publishing article...") } else { fmt.Println("Article saved as Draft!") } vals := flag.Args() fmt.Println(vals) } } ``` ```bash $ go run flag.go -p 80 -dir node_mods 1234 80 node_mods dumps 2 1 false Article saved as Draft! [1234] ``` As we can it performs the same task, but we have to use pointers as references to the arguments instead of storing them in an actual memory address. We have performed the same set of operations on the arguments and flags as we do with the other examples. We first, use the `Int` method or other methods appropriate that `String` can be used in general use cases, the function returns a reference (memory address) of the actual stored value of the arguments/flag. We can access the value from its memory address using the `*` operator. We have covered the [pointer](https://www.meetgor.com/golang-pointers/) arithmetic in the last part of the series. When we use `*port` we get the value from the memory address and thereby we can use it for the required task in the program, we can also store a copy of the variable by creating a new variable with the value of that argument. ### Parse Arguments So, if we want to parse flags, with a single value, we have seen the use of the [flag.Args](https://pkg.go.dev/flag#Args) function to get the values of the arguments passed from the command line which don't have any flag labels attached to them(just raw arguments from the CMD). Just as we used the `os.Args` variable but this function is much clean and filtered out the path to the program argument. So we can directly have the arguments which are clearly passed by the user from the command line. ```go package main import ( "flag" "fmt" ) func main() { var port int flag.IntVar(&port, "p", 8000, "Provide a port number") flag.Parse() fmt.Println(port) vals := flag.Args() fmt.Println(vals) } ``` ```bash $ go run flag.go -p 8123 8123 [] $ go run flag.go -p 8123 1234 hello true 8123 [1234 hello true] $ go run flag.go -p 8123 1234 hello true -p 9823 world 8123 [1234 hello true -p 9823 world] ``` In the above example, we can see that we have used a few non-flagged arguments from the command line. The return value of the `Args` function is a slice of string, we can then convert it into appropriate types using type casting and functions. Once the flagged arguments are parsed, if we use the `Args` function, it won't be possible to again use flagged arguments in the command line. It will be considered a simple string thereafter. That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/mr-destructive/100-days-of-golang/tree/main/scripts/files/write/) GitHub repository. ### Get Help text with PrintDefaults We can use the [flag.PrintDefaults](https://pkg.go.dev/flag#PrintDefaults) method for just printing the default values and the help text for the expected arguments from the command line in the script. We can simply use it as a help flag or use it in error messages for guiding the user to the proper arguments and flags. ```go package main import ( "flag" "fmt" ) func main() { var port int var help bool flag.IntVar(&port, "p", 8000, "Provide a port number") flag.BoolVar(&help, "help", false, "Help") flag.Parse() if help { flag.PrintDefaults() } else { fmt.Println(port) vals := flag.Args() fmt.Println(vals) } } ``` ```bash $ go run help.go -h Usage of /tmp/go-build121267600/b001/exe/help: -help Help -p int Provide a port number (default 8000) $ go run help.go 8000 [] ``` So, we can see the `PrintDefaults` function will simply print the helper text for the flags expected in the script and the default value of those flags as well. This can be used to provide a good user-friendly interface for a simple terminal application. ### Get the number of arguments We can use the [NFlag](https://pkg.go.dev/flag#NFlag) method in the `flag` package. The function returns an integer that indicates a count of the arguments that have been set from the command line. ```go package main import ( "flag" "fmt" ) func main() { var port int var dir string var publish bool flag.IntVar(&port, "p", 8000, "Provide a port number") flag.StringVar(&dir, "dir", "output_dir", "Directory") flag.Parse() fmt.Println(port) fmt.Println(dir) fmt.Println(flag.NFlag()) } ``` ```bash $ go run flag.go 8000 output_dir 0 $ go run flag.go -p 8080 8999 false hello 8080 output_dir 1 $ go run flag.go -p 8080 -dir dumps hello 1234 8080 dumps 2 ``` The `port` flag has been set from the command line, so we just have one argument set, hence the function `NFlag` returns `1` as the number of set flags. Also, the [NArg](https://pkg.go.dev/flag#NArg) method will return an integer that will count the number of arguments that have been provided leaving out the flag arguments. ```go package main import ( "flag" "fmt" ) func main() { var port int var dir string var publish bool flag.IntVar(&port, "p", 8000, "Provide a port number") flag.StringVar(&dir, "dir", "output_dir", "Directory") flag.Parse() fmt.Println(port) fmt.Println(dir) fmt.Println(flag.NArg()) } ``` ```bash $ go run flag.go 1234 8000 output_dir 1 $ go run flag.go -p 8080 -dir dumps hello 1234 8080 dumps 2 $ go run flag.go -p 8080 hello 1234 false 8080 dumps 3 ``` In the first example, we don't have any flag arguments set, we just have one unflagged argument as `1234`, hence the `NArg` function returns `1`. The second example has 2 values that are not flagged, we have set the values of `port` and `dir` as `8080` and `dumps` respectively, so the remaining unflagged values are `hello` and `1234` hence the return value as `2`. The third example has 3 unflagged values as `hello 1234 false`, hence we return `3`. That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/mr-destructive/100-days-of-golang/tree/main/scripts/cmd-args/) GitHub repository. ## Conclusion We have seen how to parse command line arguments in golang with the `os` and the `flag` packages. Though these two are not the only options for building CLI applications, they provide a clean and easy-to-start approach, also they come with the standard library which makes it even better as we don't have to mingle with third-party libraries. We saw the basics of parsing flags and arguments from a command line program. Thank you for reading. If you have any queries, questions, or feedback, you can let me know in the discussion below or on my social handles. Happy Coding :) https://www.meetgor.com/posts/2022-review Introduction Well, what a year! If 2021 was a start, 2022 was a year to convert a start into a goal. From collaborating on open-source projects to getting an in Sat, 31 Dec 2022 00:00:00 UTC ## Introduction Well, what a year! If 2021 was a start, 2022 was a year to convert a start into a goal. From collaborating on open-source projects to getting an internship, it was an exciting year! Created a couple dozen of projects(not finished as usual), wrote 40+ articles, did a couple of streams, learned the basics of golang, and spend 6 months as a backend dev intern at [Solarlabs](https://thesolarlabs.com/), just a year that was awaited for the hard work over the span of 2 years. This was a year, where I did things that were above my expectations, did an internship with college side by side for 6 months, started to stream on twitch(2 streams :), Some technical things I did and learnt in 2022: * Markata and blog revamp * Learnt Docker * Django Rest Framework Project and Django Basics completed 4 articles * Podevcast(hashnode/netlify hackathon) * Web App from scratch - django collaboration with mark gibney * Daily.dev hackathon - bookmark CLI runner up * 100 days of golang init 13 articles in April * Django deployment (Heroku, railway, nixpacks) articles * Switched from Windows 7 to Ubuntu * Bought meetgor.com and started applying for an internship * Ding 3 rounds of interviews got in at SolarLabs! * Learned Flask, pytests, drf, in the process of internship assignments at other companies * Switched to Lua config in Neovim with LSP * Created lots of projects(vacation) dj-notes, aptui, rattlesnake, django mermaid * HTMX with django, golang learning with Graphql, and REST API * Picking up with neovim plugins for django, markrunner, black snipper * deepgram and steampipe contribution at hacktoberfest * Djankata, pykyll and some other contributions to golang * Revamped APTUI with textual 2.0 * Devjournal with pykyll (bare bones ssg in python) * All in all, golang and python deep dives this year with tons of projects and django huge learning experience at the internship. ## How did it start? The start was exciting and full of expectations due to a great start in 2021. I wrote around 80 articles starting almost halfway through the year. Also started to learn Django and shell scripting. With that knowledge, I created dozens of projects like web apps for silly games, portfolios, newsletters, and shell scripts for web scrapping and framework setups. I kept on with the pace and wrote articles and made projects. I got a collaboration on an awesome open-source project \[web-app-from-scratch\]([https://github.com/no-gravity/web\_app\_from\_scratch](https://github.com/no-gravity/web_app_from_scratch)) for a docker image for setting up Django from scratch. That was a great learning experience as I didn't know docker and in February 2022 was using Windows 7 back then, bad old days. I researched and got docker up and running on a cloud instance and learned the basics of docker. ## Switch from Windows to Linux In May came a turning point for good, had to forcefully switch to Linux! I was under a ransomware attack due to the security vulnerability of windows 7 and had messed up all the files on the system. I had to completely wipe out the system and installed fresh ubuntu 22.04 on 14th May. Things changed and I was happy than ever to use my system. Thanks to that virus(xcvf ransomware virus) and natural instincts. I loved to code in the terminal, I have been using termux in lockdown which is a Linux-like terminal emulator for android. I learned the basics of bash and Linux commands with that and it was becoming my second nature to use Linux commands, so using windows was really frustrating. The incident happened at a good time and was able to switch from windows 7 to ubuntu. Setup Vim, tmux, and custom aliases was a great productive environment for me compared to windows. It changed the speed and mindset of my programming world. ## Started learning golang I started to learn golang at the end of March and created a [100-day golang series](https://www.meetgor.com/series/100-days-of-golang) on April 1st. It was not consecutive days you know, so it still looms around 25 days. It looks a lot less than I expected. Still, I feel really confident in writing golang, so much so that I decided to do the advent of code in golang this year. I completed around 13 articles in the month of April and then due to end-semester things getting in the way, was not able to keep a consecutive streak. Still, each week, I tried to write an article. And I have just 25 articles till now. It does not mean I did not learn golang, without writing any articles, I have a couple of projects on golang for creating bare-bones REST APIs and also a bit of GraphQL with DB connections. I haven't got a good amount of time to write detailed articles about them in a while, so maybe January is the time. I would like to complete 100 articles in the first quarter of 2023 that's the plan or at least the basics of web development with golang(backend of course). It was a fantastic learning experience by creating examples, and snippets and playing with some low-level stuff in the language. ## Applied for internships At the end of May, that is when my exams were ending, I started to apply for internships targeting a backend(Django stack) for maybe 3-6 months. And it was a sad time in the first few weeks. I got a few offers talking but ended in rejection as I was not able to work full-time :( That hurts after doing some assignments and an interview. ## TheSolarLabs Internship It all went on for 3 weeks, until the morning on 27 June, I read the application as "Rejected" to [SolarLabs](https://thesolarlabs.com/). Anyone would have lost hope there, and so did I. And to anyone's surprise or credit for their blunder, I got a call from them for scheduling an interview! I am not making this up, the portal still says "rejected" and in the chats, it asks for a resume! Some nervous times. After 3 days of interviews, I finally got the offer letter of confirmation. For 6 months and a good bit of stipend. It was my preferred Django stack, with some unknown grounds of the Django rest framework. I realized rather quickly in the first interview itself, I need to get some DRF stuff up! There was a lot of stuff to learn that I didn't know existed, I was baffled by how great was DRF then. That's the first bit of learning from the interview itself, it showed, how much there is to learn. Soon after the onboarding process and getting a bit familiar with the designing product, I was presented with the code repository and time to get familiarity and set up the system (python3.7, postman, editor, etc). There came the first task soon in the following week. it looked doable and indeed just a matter of time to get familiarity with the problem statement. it was a task for fetching some data from weather API and store in a specific format. As the days passed, I was introduced to the tech scrum, like a daily update for the tas done yesterday and todo today. It was really new to me, I have never heard of such systems but realized it might be a development process. Soon college started and it was a mess thereafter. had to balance college attendance and attend scrum with clashing times sometimes. But to date got off with the semesters. ### Key takeaways from the internship I don't want to get into all the details of the internship tasks but here's a summary of what I learned in a gist. * Django Rest Framework viewsets, routers, and actions * Django Signals, Admin section, API designs in a better way * Azure (some specific things like function deployment, config, sas tokens, pipelines, python SDK, etc.) * SQL ( got some good practice on joins and nested queries) * ERP technology(erpnext and python setup) * JWT tokens set up in Django * Process for designing API from scratch for a given feature * Cashfree and Stripe integration in Django * Dockerfile and buildkit specific features ## Learning Constantly Currently in a hope of continuing the internship as I have thoroughly enjoyed working here and it has been a great learning experience. But one thing was really important this year again, to not stop learning and building. Don't mind if the idea is small or useless, just building the project and overcoming the entry barrier for learning the technology has been the quote for everything in my developer journey. > When your mind feels lost, letting it free and driving new places strikes a inspriation for another take off, don't run, enjoy the walk, it's a marathon not a sprint. Building projects which are useless or not making sense to anybody might be a different thing but what it means to yourself matters the most. There might be instances where you might be failing, not having any ideas, burnout, imposter syndrome, and some other fuzzy words in tech. These situations are meant to improve you in one or the other way, If you run out of ideas, just take a break, look at your personal state and hobbies besides programming and get some inspiration from other fields and apply them in the programming world were some lessons I learned in some rough times during the year. There will be or rather should be times when you feel down or lose inspiration, these times drive the minds in different directions as programming can be a bit draining and one can quickly lose motivation if there are no results straight away. > Hard work takes time to turn into results, patience and actions are the key. I had a hobby for writing, I had written a novel(maybe not a novel it's not that big) when I was in 8th grade, and had a ton of ideas for stories but was stuck with the hypothetical thing called the perfection of time, plot, and ideas. I watched a youtube video about writing and it just fascinated me to write again, and it began. I started to write my novel which is handwritten into an ebook. I was consistent for a few days but things got wild and were moved away from it. The new year will be the perfect free time to go for it and at least have one fiction novel to my name. I am also reading fantasy novels, I have read Elantris and am in mid of reading Mistborn. I have been enjoying reading and writing as a programmer, but want to get into fantasy as well. It is so exciting and fascinating as a human to fly with creativity. I am grateful to have eyes and a healthy mind to read and see the beauty of the world. ## Hits and Misses in 2022 There were a few pullbacks in 2022 for me, if there are good times, one must be prepared for bad ones it seemed. The worst thing about bad times is that you sometimes don't realize that you are in one. I had almost the same count of articles compared to the previous year comparing a whole year to half the year(I started blogging in June 2022). * Unable to write out words * Feeling exhausted and anxious sometimes * Not able to communicate properly with mentors * Sometimes, college and working on internship tasks clash badly * addicted to solving a problem and can't focus on other things(maybe a good thing sometimes) ## Tech list for 2023 * Golang deep dive with projects and open-source contributions * Machine Learning (Tensorflow python and Keras) * Kafka * Kubernetes and Microservices * AoC with Rust So, these are some of the technologies I will try to learn in 2023. I have a basic knowledge of golang, but I will like to get into the specific things golang is used for like concurrency and channels. I have been wanting to learn machine learning with projects, and this year I am serious about it, I have a couple of ideas for web apps with ML, and will try to get it up and rolling. Some specific technologies I want to learn to include Kafka for data pipelines and streams. Some deployment tools like Kubernetes but before that, I will be learning a bit more about docker networking. This year I did AoC with golang, to complete all 25 days, but did around 12 days. Still will be trying to solve this over the month. Next year, I will be trying AoC in Rust. I love low-level programming languages, go is great but rust is extreme. ## Conclusion So, that's a wrap for 2022. A great year for learning a lot of stuff. Hopefully, the next year will be a better one. More collaboration and more opportunities. I am grateful to so many people who came into my life and have played important roles for me to develop into a backend developer and even a human. My mentors and colleagues, real-life and Twitter friends, and GitHub collaborators. I would have not reached here if I would have not met any one of them. Thank you. May you too have a great year and life ahead full of opportunities and success. Happy New Year, Happy Coding :) https://www.meetgor.com/posts/golang-file-write Introduction In the 24th post of the series, we will be taking a look at how we can perform write operations to a file using golang. We will be using the packag Sun, 18 Dec 2022 00:00:00 UTC ## Introduction In the 24th post of the series, we will be taking a look at how we can perform write operations to a file using golang. We will be using the `os` package in most operations along with `bufio` text manipulations. We will be performing write operations like appending, deleting, and replacing a file using golang. We will be heavily leveraging standard library packages like `os`, `bufio`, `bytes` and `fmt`. We will also be looking into overwriting and string formatting to a file. ## Write to a File The first part of this section is the basic write operation to a file, we assume we are writing to a fresh file and overriding the contents of the existing file. The next section will cover the appending of content to the file and so on. In this example, we will see how we perform basic writing operations to write a string, a slice of string to a file. ```go package main import ( "log" "os" ) func HandleError(err){ if err != nil { log.Fatal(err) } } func main() { str := "Hi, I am a gopher! " f, err := os.OpenFile("abc.txt", os.O_WRONLY, 0660) // f, err := os.Create("abc.txt") HandleError(err) _, err = f.Write([]byte(str)) HandleError(err) defer f.Close() } ``` ```bash $ cat abc.txt $ go run main.go $ cat abc.txt Hi, I am a gopher! ``` So, we have used a simple golang script to write to a file that exists/has already been created. If you don't want errors while having to write on a file that does not exist, use the [Create](https://pkg.go.dev/os#Create) method instead which is similar to the `Open` method but creates a file if it doesn't exist. We use the [Write](https://pkg.go.dev/os#File.Write) method to overwrite contents to the file, it takes in a parameter as a slice of byte, so we typecast the string `str` into `[]byte` using the `[]byte(str)` syntax. Thereby we write the contents of the string into the file. We use the defer keyword for closing the file at the end of the script or the end of the main function scope. ### Write slice of strings to file We can even write slice of string to file using a for loop and appending each string with a new line character. ```go package main import ( "log" "os" ) func HandleError(err){ if err != nil { log.Fatal(err) } } func main() { f, err := os.Create("abc.txt") //f, err := os.Open("abc.txt", os.O_WRONLY, 0660) langs := []string{"golang", "python", "rust", "javascript", "ruby"} for _, lang := range langs { _, err := f.WriteString(lang + " ") HandleError(err) } defer f.Close() } ``` ```plaintext $ cat abc.txt $ go run main.go $ cat abc.txt golang python rust javascript ruby ``` We have used the [WriteString](https://pkg.go.dev/os#File.WriteString) method which will take in a string as a parameter instead of a slice of bytes. So, we don't have to type cast into slice of bytes. So, as we can see we have written the string slice into a file. ### Over Write The minimal code to write to a file is the [WriteFile](https://pkg.go.dev/os#WriteFile) function in the [os](https://pkg.go.dev/os) package, it overrides the content of the file with the provided slice of bytes, the name of the file, and the necessary permission to write. The funciton additionally creates a file if it does not exist, which is one less reason for the error. Though it returns an error object, the error might be created due to not right permissions to write to the file, encoding issues, etc. ```go package main import ( "log" "os" ) func main() { data := []byte{115, 111, 109, 101, 65} err := os.WriteFile("test.txt", data, 0660) log.Println(err) s := "Hello" err = os.WriteFile("test.txt", []byte(s), 0660) log.Println(err) } ``` ```plaintext $ go run main.go 2022/12/17 19:24:13 <nil> 2022/12/17 19:24:13 <nil> $ cat test.txt Hello ``` So, we have used the `WriteFile` method two times in the script, it first takes in a slice of bytes as it is defined as `data` which corresponds to `115 -> s`, `111 -> o`, `65 -> A`, ASCII mapped to strings. The slice of bytes can be taken as a string like `someA` as the literal value of the underlying slice of the byte. So, we take that slice of byte and parse it to the second parameter of the WriteFile function. The first parameter is a string path of the file we want to write the contents to, the third parameter is the file permission. We have set it as `0660` indicating read(4) + write(2) to the group and the user and no permission to the other users. The function will return an error if any, or else it simply overwrites the data in the file. In this case, we have called the `WriteFile` method with string `s` type cast to slice of bytes at the end of the script so we see the file has contents as `Hello` instead of `someA`. If we reverse the action, we don't see the `Hello` string in the file. ```go package main import ( "log" "os" ) func main() { s := "Hello" err := os.WriteFile("test.txt", []byte(s), 0660) log.Println(err) data := []byte{115, 111, 109, 101, 65} err = os.WriteFile("test.txt", data, 0660) log.Println(err) } ``` ```plaintext $ go run main.go 2022/12/17 19:24:13 <nil> 2022/12/17 19:24:13 <nil> $ cat test.txt someA ``` As we can see the `Hello` has been overwritten by `someA`. ### Write formatted string We can even use fmt to write formatted strings to a file. Just like we can take inputs with `Scanf`, we can use [Fprint](https://pkg.go.dev/fmt#Fprint) and other similar functions like [Fprintf](https://pkg.go.dev/fmt#Fprintf), and [Fprintln](https://pkg.go.dev/fmt#Fprintln) functions to print/add contents to the file. ```go package main import ( "fmt" "log" "os" ) func HandleErr(err error) { if err != nil { log.Fatal(err) } } func main() { f, err := os.Create("temp.txt") HandleErr(err) name, lang, exp := "John", "go", 2 _, err = fmt.Fprint(f, "Hi, I am ", name, " ") HandleErr(err) _, err = fmt.Fprintf(f, "Language of choice: %s. ", lang) HandleErr(err) _, err = fmt.Fprintln(f, "Having", exp, "years of experience.") HandleErr(err) defer f.Close() } ``` ```plaintext $ cat temp.txt cat: temp.txt: No such file or directory $ go run format.go $ cat test.txt Hi, I am John. Language of choice: go. Having 2 years of experience. ``` So, we can see that we have used all three methods having their own use cases, we can use `Fprint` for simple strings, `Fprintf` for formatting the block of a string with multiple placeholders, and the `Fprintln` which works simply like `Fprint` but it adds a new line itself, we don't need to specify it explicitly. ### Append If we want to append text to a file, we can use the [OpenFile](https://pkg.go.dev/os#OpenFile) function and provide a few parameters to append the contents instead of overwriting. Here, we have two steps, open the file and then write the contents in the file. So while opening the file, we provide a few options as parameters to make the fine-tuned system call like only open for read, write or append modes. These options are defined as constant int values in the [os package](https://pkg.go.dev/os#pkg-constants). ```go package main import ( "log" "os" ) func HandleError(err error) { if err != nil { log.Fatal(err) } } func main() { s := "Hello" err := os.WriteFile("test.txt", []byte(s), 0660) HandleError(err) s = "World" f, err := os.OpenFile("test.txt", os.O_APPEND|os.O_WRONLY, 0660) HandleError(err) _, err = f.WriteString(s) HandleError(err) defer f.Close() } ``` ```plaintext $ go run main.go $ cat test.txt HelloWorld ``` So, from the above example, we are able to append text into a file, we have first added the `Hello` world string into the file using the `WriteFile` method to indicate we overwrite the previous contents of the file. We then use the [OpenFile](https://pkg.go.dev/os#OpenFile) method to open a file provided in the first parameter as a string path. The second parameter is the options to be passed for performing operations on the opened file, we always have to use them `defer` to close the file or other resource-locking operations. We have specified the `os.O_WRONLY` and the `os.O_APPEND` options indicating we want to write to the file while the file is open and specifically append to the file. So this is fine-tuning the opened file operation. We can use the ReadFile or WriteFile operation which is just used for simple read and write operations respectively. We use the [WriteString](https://pkg.go.dev/os#File.WriteString) method, but we can even use the [Write](https://pkg.go.dev/os#File.Write) method to write a slice of byte instead. This is just used for exploring the different options in the file types of the os package. ### Append at a specific line We can also add content to a specific line or a portion of the file. There are no direct functions in golang to do the same, we will have to do some manual fine-tuning of file operations to append a particular text at a specific line. ```go package main import ( "bufio" "bytes" "log" "os" ) func HandleError(err error) { if err != nil { log.Fatal(err) } } func main() { /* test.txt Hi Hello World Gopher */ f, err = os.OpenFile("test.txt", os.O_RDWR, 0660) defer f.Close() HandleError(err) m := bufio.NewScanner(f) bytes_till := 0 // line to be appended line_till := 2 var lines_after string var lines_till string i := 0 for m.Scan() { line := m.Text() if i < line_till { bytes_till += bytes.Count([]byte(line), []byte{}) if i > 0 { lines_till += " " } lines_till += line } else { lines_after += " " + line } i += 1 } HandleError(m.Err()) insert_text := lines_till + " Inserted content" insert_text_bytes := bytes.Count([]byte(insert_text), []byte{}) lines_after_bytes := bytes.Count([]byte(lines_after), []byte{}) err = os.WriteFile("test.txt", []byte(insert_text), 0660) HandleError(err) _, err = f.WriteAt([]byte(lines_after), int64(lines_after_bytes)+int64(insert_text_bytes)) HandleError(err) /* test.txt Hi Hello Inserted content World Gopher */ } ``` ```plaintext $ cat test.txt Hi Hello World Gophers $ go run append.go $ cat test.txt Hi Hello Inserted content World Gophers ``` We have inserted `Inserted content` after the second line because the `line_till` the variable is set to `2` So, in the above example, we have used a bunch of packages to append a string or any form of text to a particular line. We first read the contents of the file, by using the `OpenFile` method which will open the file with certain permissions. We need to close the file at the end of the script so we simply use the `defer` keyword before the `f.Close()` method call. We then start to scan the file buffer, by creating a `Scanner` object with the `NewScanner` method. Then, with the scanner object of the file content, we then can use the `Scan()` method to scan the file contents line by line. By converting the content at each line from a slice of bytes to a string using the `Text`, we append to a string `line`, this will be used for keeping the count of bytes for appending text before the newly inserted text. The `line_till` variable is used for the line number from which we want to append to the text after. We count the bytes for the current line and add it to the `bytes_till` variable indicating the number of bytes there before appending content. We have a simple if-else check for the first line that is for appending a new line of characters. We append the lines into a single string `lines_till`. The string `insert_text` is created by appending all the lines before the line number `line_till` with the actual content to be inserted. We calculate the number of bytes using the [Count](https://pkg.go.dev/bytes#Count) method in the bytes package. The separator is kept blank. The `lines_after` is also been created as a single string of lines after the line number in the file. We add the `insert_text` (lines before + inserted text) into the file using the `WriteFile` which will override the contents of the file. Then we append the `lines_after` string as a slice of bytes to the `insert_text_bytes + lines_after_bytes` so we get the byte number position to append the `lines_after` string. In short, we basically overwrite the file by creating two strings (slice of bytes) one which has the lines before the line number with the text to be inserted and the second string has all the lines after the line number. ## Replace text in a file Using the [bytes.Replace](https://pkg.go.dev/bytes#Replace) method, we can first read all the bytes and replace the old with the new text, and store them as a slice of bytes. We then write these slices of bytes to the file again, so we first read the contents into slices of bytes, replace the content of the byte and then overwrite the contents with the slice of bytes. It's quite simple. ```go package main import ( "bytes" "log" "os" ) func HandleError(err error) { if err != nil { log.Fatal(err) } } func main() { filename := "test.txt" file, err := os.ReadFile(filename) HandleError(err) old_text := "Hello World" new_text := "Bye" new_content := bytes.Replace(file, []byte(old_text), []byte(new_text), -1) err = os.WriteFile(filename, new_content, 0660) HandleError(err) } ``` ```plaintext $ cat test.txt Hi Hello World Gophers $ go run main.go $ cat test.txt Hi Bye Gophers ``` As we can see we have replaced `Hello World` it with `Bye`. So, the \[Replace\] method in the bytes package takes in the parameters like the slice of bytes which should be the actual contents of the file, the old text to be replaced with again as a slice of bytes, and the new text to replace also as the slice of bytes, the final parameter is the number of replacements to be made. Here `-1` indicates there are no limits on how many replacements can be done, it can be `1`, `2` for replacing the first n occurrence of the old text, depending on how many times you want to replace the content in the file. ## Delete Text from a File We can use the [os.Truncate](https://pkg.go.dev/os#File.Truncate) method to delete the contents of the file. The `Truncate` method takes in the parameters like the file path string and the size of the file to truncate or set to. If we set the second parameter to `0`, the file size will be zero and all the contents will be deleted or removed. ```go package main import ( "log" "os" ) func main() { /* test.txt Hi Hello World Gophers */ err := os.Truncate("test.txt", 0) if err != nil { log.Fatal(err) } /* test.txt is empty */ } ``` ```plaintext $ cat test.txt Hi Hello World Gophers $ go run delete.go $ cat test.txt ``` As we can see that the contents of the file are emptied if we set the second parameter(size) of the `Truncate` method as 0. We can also set the value of the size as the number of bytes to keep, so instead of `0` we can set it to `n` positive integer to only save the first `n` bytes in the file. ```go package main import ( "log" "os" ) func main() { /* test.txt Hi Hello World Gophers */ err := os.Truncate("test.txt", 6) if err != nil { log.Fatal(err) } /* test.txt Hi Hel */ } ``` ```plaintext $ cat test.txt Hi Hello World Gophers $ go run delete.go $ cat test.txt Hi Hel ``` So, in the above example, if we set the size parameter to the Truncate method as `6`, it will keep the size of the file to 6 bytes. So we only see `Hi Hel`, the new line is a single byte. The rest of the content is deleted. This is how we previously deleted all the bytes from the file by setting the size to `0`. That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/mr-destructive/100-days-of-golang/tree/main/scripts/files/write/) GitHub repository. ## Conclusion So, from this section of the series, we were able to perform write operations on a file using golang. We used the packages from the standard library and performed operations like write, append, overwrite, delete and replace to a simple text file, but it could have been any file format. Thank you for reading. If you have any queries, questions, or feedback, you can let me know in the discussion below or on my social handles. Happy Coding :) https://www.meetgor.com/posts/golang-config-file-read Reading specific file types (JSON, YAML, TOML) In the previous post, we have seen how to read files in golang, in this extended post of that part, we will look Tue, 01 Nov 2022 00:00:00 UTC ## Reading specific file types (JSON, YAML, TOML) In the previous post, we have seen how to read files in golang, in this extended post of that part, we will look into reading some specific files used for configuration and storing data like JSON, YAML, TOML, CSV, etc. We will see how to read files and get individual pieces in the files. We'll use packages like `os`, `ioutil` and `encoding` to perform reading operations on file and file objects. ### Reading a JSON File Golang has built-in support for reading JSON files, but still, we can and need to have low-level controls on how to parse and extract content from the file. Let's say we have a `json` file named `blog.json`, we can use the [encoding/json](https://pkg.go.dev/encoding/json) package to convert the JSON data into a GO object (an object that is native and understandable to go). The [Unmarshal](https://pkg.go.dev/encoding/json#Unmarshal) function is used to convert the slice of bytes from the file, into a map object. ```json { "title": "Golang Blog Series", "date": "22nd October 2022", "tags": ["go", "files"], "words": 1500, "published": true } ``` The above is a simple JSON file, this file has a few types of key-value pairs like string, list, integer, and boolean. But we can also have nested objects and a list of those nested objects. ```go package main import ( "encoding/json" "log" "os" ) func main() { f, err := os.ReadFile("blog.json") if err != nil { log.Println(err) } var data map[string]interface{} json.Unmarshal([]byte(f), &data) log.Println(data) for k, v := range data { log.Println(k, ":", v) } } ``` **I have removed the time stamp from the logs below so as to clearly see the output. We can use `fmt` to print the simple things while keeping consistent with the rest of the snippets in the series.** ``` $ go run json.go map[date:22nd October 2022 published:true tags:[go files] title:Golang Blog Series words:1500] published : true title : Golang Blog Series date : 22nd October 2022 tags : [go files] words : 1500 ``` The file is read using the [os.ReadFile](https://pkg.go.dev/os#ReadFile) method, that takes in a string as a path to the file and returns a slice of bytes or an error if there was an issue in reading the file. The parsed slice of byte is than passed as the first argument to the `Unmarshal` method in the `encoding/json` package. The second parameter is the output reference where the parsed JSON will be stored or returned. The function does not return the parsed content instead returns an error if there arose any while parsing the JSON content. As we can see we have got a map of `string` with an `interface`. The interface is used because the value of the key can be anything. There is no fixed value like a `string`, `int`, `bool`, or a nested `map`, `slice`. Hence we have mapped the JSON object as a map of `string` with an `interface`. The type of the value is identified with the interface it has attached to it. Let's take a look what is the type of each value in the map. ``` published : true bool title : Golang Blog Series string date : 22nd October 2022 string tags : [go files] []interface {} words : 1500 float64 ``` Here, we can see it has correctly identified the string type of the fields like bool in case of true or false, a string for string type of values, the fourth field however has a list interface attached to it. The default precedence for `float64` over integer is the reason the `1500` value is of type `float64`. ### Reading a YAML File Though there is no standard package for parsing/unmarshaling `YAML` files in golang, it's quite easy to use a third-party package and use it to read YAML files. The package [gopkg.in/yaml.v3](https://pkg.go.dev/gopkg.in/yaml.v3) is used for encoding and decoding YAML files. We'll be just using it for decoding a YAML file by reading it and converting the file object into native Go objects like maps, lists, strings, etc. The below steps can be used for setting up the package and installing the YAML package locally. ``` go mod init <your_project_package_name> go get gopkg.in/yaml.v3 ``` This should create two files namely `go.mod` and `go.sum` with the dependency of the `gopkg.in/yaml.v3` package. ```yml title: "Golang Blog Series" date: 2022-11-01 tags: ["go", "files"] published: false words: 1500 ``` The above file is a simple YAML config, we'll follow similar kind of examples for the dummy files used in the examples. ```go package main import ( "log" "os" yaml "gopkg.in/yaml.v3" ) func main() { f, err := os.ReadFile("blog.yaml") if err != nil { log.Fatal(err) } var data map[string]interface{} err = yaml.Unmarshal(f, &data) if err != nil { log.Fatal(err) } log.Println(data) for k, v := range data { log.Println(k, ":", v) } } ``` ``` $ go run yaml.go map[date:22nd October 2022 published:false tags:[go files] title:Golang Blog Series words:1500] published : false words : 1500 title : Golang Blog Series date : 22nd October 2022 tags : [go files] ``` The above code and output demonstrate the usage of the `yaml.v3` package for reading a YAML file. Firstly, we read the file into a single-string object with the [os.ReadFile()](https://pkg.go.dev/os#ReadFile) method. The method will return a `[]byte` (slice of byte) or an error. If there is an error, we simply log or panic out of the program, else we can use the [yaml.Unmarshal](https://pkg.go.dev/gopkg.in/yaml.v3#Unmarshal) method to convert the string/slice of the byte into a map or a pre-defined struct. In this example, we just keep things simple by storing the file content as `map [string, interface{}]`, i.e. a map of `string` and an `interface`. The key for YAML can be only a string or an integer. It can't have unrestricted data types like the value can have. Though if you want to be unrestrictive, you can use a map of `map[interface{}]interface{}` to make the key any shape you like to have. So, we have created a variable called `data` as a map of `string` and `interface{}`, basically key can be a string and the value can be any type of interface depending on the parsed literally from the file object. The `Unmarshal` function takes in two parameters, the first being the slice of byte i.e. the file contents, and the second being the output variable. Now, the method does not return the parsed YAML, it can return an error if there arose, so we need to set the second parameter as a pointer to the object into which we want to store the parsed YAML. In the example, we have called `Unmarshal(f, &data)` which will fetch the contents from the slice of bytes `f` and output the parsed YAML from the slice of bytes into the memory location of `data` and hence using `&data` indicating the pointer to the variable(fetch the memory address). So, that is how we obtain the map of keys and values from the YAML config, thereafter, you can iterate on the map, access the keys and values, type caste them as per requirement, and basically have control over what processing needs to be done to the parsed YAML content. ``` published : false bool words : 1500 int title : Golang Blog Series string date : 22nd October 2022 string tags : [go files] []interface {} ``` I have just printed the types of the values in the above output as `log.Printf("%T", v)`, we can see the types are being correctly recognized and being parsed. The last object is indeed a `slice` so it has an interface of the slice(array) attached to it. ### Reading a TOML file YAML and TOML are almost identical, though toml has more restrictive configuration options, but is more readable than YAML, as YAML can get complicated pretty quickly. Though both of them have their pros and cons, YAML is used everywhere in the DevOps world, configs, whereas TOML is the format of choice for python packaging, and static site generation configs. Let's see how we can use golang to read TOML files. ``` $ go mod init <your_project_package_name> $ go get github.com/pelletier/go-toml ``` The above commands are used for setting up a golang package or project and installing the [go-toml](https://pkg.go.dev/github.com/pelletier/go-toml) package. Once the above commands are done executing, it will generate `go.mod` and `go.sum` files used for storing dependencies and packages installed for the project locally. ```toml [blog] name='techstructive-blog' tags=['go','django','vim'] author='meet gor' active=true [author] name='Meet Gor' github='mr-destructive' twitter='meetgor21' posts=80 ``` The above is the sample file `blog.toml` which we will use to read in the go script below. The toml file has a similar structure as we have seen in the previous examples. We have different data types like string, boolean, integer, and list. ```go package main import ( "log" "os" toml "github.com/pelletier/go-toml" ) func main() { f, err := os.ReadFile("blog.toml") if err != nil { log.Fatal(err) } var data map[interface{}]interface{} err = toml.Unmarshal(f, &data) log.Println(data) if err != nil { log.Fatal(err) } for k, v := range data { log.Println(k, ":", v) switch t := v.(type) { case map[string]interface{}: for a, b := range t { log.Println(a, ":", b) } } } } ``` ``` $ go run toml.go map[author:map[github:mr-destructive name:Meet Gor posts:80 twitter:meetgor21] blog:map[active:true author:meet gor name:techstructive-blog tags:[go django vim]]] blog : map[active:true author:meet gor name:techstructive-blog tags:[go django vim]] name : techstructive-blog tags : [go django vim] author : meet gor active : true author : map[github:mr-destructive name:Meet Gor posts:80 twitter:meetgor21] name : Meet Gor github : mr-destructive twitter : meetgor21 posts : 80 ``` So, in the above example and output, the YAML file was read and the key-value pairs inside them were read. The first thing we do is read the file `blog.toml` with `ioutil` package, with the `ReadFile` function. The function takes in the string as the path to the file to be read and returns a slice of byte. We use this slice of byte as a parameter to the [Unmarshal](https://pkg.go.dev/github.com/pelletier/go-toml#Unmarshal) method. The second paramter for the `Unmarshal` is the output variable(usually a pointer to a variable), we have created a map of `interface{]` with an `interface` as we see there can be nested keys which hold the name of the config. The variable `data` is a map of `interface{}` to an `interface{}`, and we parse the memory address to the `data` variable to the `Unmarshal` method. Thereby the parsed `TOML` content is stored in the data variable. ``` name : techstructive-blog string tags : [go django vim] []interface{} author : meet gor string active : true bool name : Meet Gor string github : mr-destructive string twitter : meetgor21 string posts : 80 int64 ``` The above is a verbose output for the type of the values as parsed by golang, we have `string`, `bool`, `int64`, and a `slice` (list with interface{} attached with it). Only types like `string`, `bool`, `int`, `float64` can be parsed from the Unmarshal function, other than these types, the type will have an interface attached to it. In such cases, where the type of value is not among the 4 types(string, bool, int float), we can use a pre-defined struct to parse the content from the file. Though it would require a strict structure and predictable response from the parsed file. ### Reading CSV file We can even read a CSV file in golang, we have seen in the previous post, we have used custom delimiters in the parsing of the file. ```csv id,name,posts,exp 21,jim,23,2 33,kevin,39,1 45,james,70,2 56,chris,89,3 ``` The above file is a sample csv file, though the size is too small, we can use it as an example. ```go package main import ( "encoding/csv" "log" "os" ) func main() { f, err := os.Open("temp.csv") check_error(err) reader := csv.NewReader(f) n, err := reader.ReadAll() check_error(err) for _, line := range n { for _, text := range line { log.Println(text) } } } ``` ``` $ go run main.go id name posts exp 21 jim 23 2 33 kevin 39 1 45 james 70 2 56 chris 89 3 ``` The CSV package has a [NewReader](https://pkg.go.dev/encoding/csv#NewReader) method that accepts an `io.Reader` and returns a `Reader` object. After parsing the reader, we use the [ReadAll](https://pkg.go.dev/encoding/csv#Reader.ReadAll) method to return a 2d string or an error if there exists an error while parsing the content. You can get a detailed explanation of the CSV parsing and reading in the [previous post](https://www.meetgor.com/golang-file-read/#Read%20File%20by%20a%20delimiter). ### Reading CSV from URL The CSV file can also be read from the URL, the content of the file is a `response.Body` in place of the file object reference, in the previous example, the [os.Open()](https://pkg.go.dev/os#Open) method returns a [os.File](https://pkg.go.dev/os#File) object. We use the `http.Get(string)` method to get the response from the URL for reading the CSV file present on the web. ```go package main import ( "encoding/csv" "log" "net/http" ) func main() { url := "https://github.com/woocommerce/woocommerce/raw/master/sample-data/sample_products.csv" response, err := http.Get(url) if err != nil { log.Println(err) return } defer response.Body.Close() reader := csv.NewReader(response.Body) n, err := reader.ReadAll() if err != nil { log.Println(err) } for _, line := range n { for _, text := range line { log.Println(text) } } } ``` ``` $ go run csv.go <feff>ID Type SKU Name Published Is featured? Visibility in catalog Short description Description Date sale price starts Date sale price ends ... ... ... ``` So, that's how we can read a CSV file from the URL. By fetching the CSV URL `https://github.com/woocommerce/woocommerce/raw/master/sample-data/sample_products.csv` from the [http.Get](https://pkg.go.dev/net/http#Client.Get) method, this will get us the [response.Body](https://pkg.go.dev/net/http#Response) that contains the actual CSV file content. The response than can be parsed to the `csv.NewReader(*Os.File).ReadAll()` i.e. [reader.ReadAll()](https://pkg.go.dev/encoding/csv#Reader.ReadAll). The function returns a multidimensional slice `[][]slice` that can be iterated and parsed as per requirement. ### Reading XML file XML is the de facto standard for RSS feeds, it is widely used in many places and is still all over the web. We'll see an example to read an XML file locally, but as we saw in the above example, we can also read an RSS link from the web. Just like CSV, we have [encoding/xml](https://pkg.go.dev/encoding/xml), and the standard library has all the functions used for parsing the XML files. We will be using a local XML file called `rss.xml`, and reading the contents from the tags in the file. ```xml <?xml version="1.0" encoding="UTF-8" ?> <channel> <title>Meet Gor</title> <description>Techstructive Blog Feed</description> <item> <title>Why and How to make and use Vim as a text editor and customizable IDE</title> <link>https://www.meetgor.com/vim-text-editor-ide</link> </item> <item> <title>Setting up Vim for Python</title> <link>https://www.meetgor.com/vim-for-python</link> </item> </channel> ``` The above example is a short part of my blog's [rss](https://www.meetgor.com/rss) feed. I have just trimmed the unwanted part and will be just using the tags that we want to fetch. ```go package main import ( "encoding/xml" "log" "os" ) type Channel struct { XMLName xml.Name `xml:"channel"` Title string `xml:"title"` Description string `xml:"description"` Item []Item `xml:"item"` } type Item struct { XMLName xml.Name `xml:"item"` Title string `xml:"title"` Link string `xml:"link"` } func check_error(err error) { if err != nil { log.Fatal(err) } } func main() { f, err := os.ReadFile("rss.xml") check_error(err) defer f.Close() d := Channel{} err = xml.Unmarshal(f, &d) check_error(err) for _, item := range d.Item { log.Println(item.Title) } } ``` ``` $ go run xml.go {{ channel} Meet Gor Techstructive Blog Feed [{{ item} Why and How to make and use Vim as a text editor and customizable IDE https://www.meetgor.com/vim-text-editor-ide} {{ item} Setting up Vim for Python https://www.meetgor.com/vim-for-python}]} Why and How to make and use Vim as a text editor and customizable IDE Setting up Vim for Python ``` The above example uses a couple of `struct` like `Channel` and `Item` that stores the tag data like `title`, `description`, `link`, etc. Unlike the JSON, YAML, and toml files, XML can't parse the content directly we need a structure to parse into. And that's fine as XML is not much dynamic in terms of config, we usually have standard tags and elements which can be pre-defined in a struct type. In this example, the RSS feed has a `Channel` tag with `title`, `description`, and `item`. **NOTE: Use Title case for the fields of the structs. It will make the fields public, I spent a few hours debugging that really :)** So, we define the `Channel` struct with fields like `Title` as a string which is a tag in the file as `xml:"title"`. This means the title in the tag of the XML will be stored in the field as a string in the attribute name as `Title`. Similarly, we have fields like `Description` and `Item[]` this is a list or multiple of `item` tags that might be present in the XML file. The `XMLName` is used for identifying the parent tag for the struct, so we use `channel` for the first struct as it is the first tag appearing of the hierarchy in the XML file. So, we create an object for the root structure as `Channel{}` (an empty object instantiated). The `xml.Unmarshal` function is parsed with the content of the file as `data` which is a slice of byte as we have seen in the previous examples. The slice of byte is then used in the `Unmarshal` method as the first parameter and the reference of an empty `Channel` object as the second parameter. The second parameter will be to store the parsed XML content from the file. I have a few examples on the GitHub repository covering the reading of files from a URL for the CSV, and XML files. But, this concept in the example, can be applied to JSON, YAML, and other file formats as well. That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/Mr-Destructive/100-days-of-golang/tree/main/scripts/files/read/config_files) GitHub repository. ## Conclusion So, that's it from this post, we covered how to read specific configuration files like `JSON`, `CSV`, `YAML`, `TOML`, and `XML`. We saw how to read a local file and also touched on the concept to read contents from a file on the web with a URL. We also saw how we can use pre-defined structs to parse content from a file, especially for XML. Thank you for reading. If you have any queries, questions, or feedback, you can let me know in the discussion below or on my social handles. Happy Coding :) https://www.meetgor.com/posts/golang-file-read Introduction In the 22nd post of the series, we will be looking into the file-handling process in golang, in the next few posts, we will cover the operations on Sun, 23 Oct 2022 00:00:00 UTC ## Introduction In the 22nd post of the series, we will be looking into the file-handling process in golang, in the next few posts, we will cover the operations on file using golang. In this first entry of the file handling sub-series, we will understand the `READ` operation with files. We will see different ways to read a file, it can be word by word, line by line, or even custom chink by chunk. While dealing with files, we will also use standard library packages such as `os`, `bufio`, etc. We'll also touch on how we can read files from a remote location. Using golang, we will have a low-level interaction with file management but golang also abstracts the most of heavy lifting and management of files for us, so it becomes quite easy to work with files. ## Read the file as a single string (using os.ReadFile) We can use the [os](https://pkg.go.dev/os) package in golang, in which we have access to the [ReadFile](https://pkg.go.dev/os#ReadFile) funciton. The `ReadFile` function takes in a parameter as a string which should be a file name, it returns a slice of bytes or an error. We have discussed the error handling in the previous part of the series. So, we have to use the comma ok error syntax to get the appropriate return value from the funciton. We can grab the slice of bytes as the text we want or an error if there are errors like a file doesn't exist, it's a folder, etc. ```go package main import ( "os" "log" ) func main() { text, err := os.ReadFile("sample.txt") if err != nil { log.Fatal(err) } log.Println(string(text)) } ``` ``` $ go run main.go 2022/10/23 22:39:11 Golang is a programming language. created: 2007 type:static ``` So, under the hood here, we have text as a slice of bytes. We can iterate over the text as a slice and get the character by character-in the content of the file. Though we don't directly interact with the file content, we are storing it in a variable. In technical words, the file is directly loaded into the memory at once. We thereby return a single string object containing the content of the file. ## Read file line by line We can even read a file line by line. Using the [bufio.NewScanner()](https://pkg.go.dev/bufio#NewScanner), the function takes in a [Reader](https://pkg.go.dev/io#Reader) object in our case it will be a file object. The function returns a scanner object that can be used to read the text with a particular scanner method. The returned object can be used in the loop to iterate over the content, in our case, we use the [Scan](https://pkg.go.dev/bufio#Scanner.Scan) method to split the file into lines. But we can use other methods like [ScanWords](https://pkg.go.dev/bufio#ScanWords) for scanning words, [ScanRunes](https://pkg.go.dev/bufio#ScanRunes) for scanning character by character, [ScanBytes](https://pkg.go.dev/bufio#ScanBytes) for scanning byte by byte. ```go package main import ( "bufio" "log" "os" ) func main() { f, err := os.Open("sample.txt") if err != nil { log.Fatal(err) } line_list := []string{} defer f.Close() scanner := bufio.NewScanner(f) for scanner.Scan() { line := scanner.Text() line_list = append(line_list, line) } if err := scanner.Err(); err != nil { log.Fatal(err) } for _, line := range line_list { log.Println(line) } } ``` ``` $ go run line.go 2022/10/23 22:39:50 Golang is a programming language. 2022/10/23 22:39:50 created: 2007 2022/10/23 22:39:50 type:static ``` In the above example, the file is read in with the `bufio.NewScanner` and is iterated line by line with the help of the `Scan` function. The text in the line is scanned and stored in the variable `line` which is a string, this is further appended to the string slice `line_list`. Hence we can iterate over the file content line by line and store the results as a string array. Here we have used the `defer` keyword before calling the `f.Close()` method because we want to close the file after performing operations on the file. The defer will call the function at almost the end of the main function i.e. at the end of the program. ## Read File by a delimiter We can even read a file with a custom delimiter which can be used to read CSV or other delimiters. The [csv](https://pkg.go.dev/encoding/csv) package has a [NewReader](https://pkg.go.dev/encoding/csv#NewReader) function which takes in a object of file content, and it will return a [Reader](https://pkg.go.dev/encoding/csv#Reader) object. We can alter the attribute `Comma` in the `Reader` object and set it to any value we want as a delimiter. Thereafter we can read the whole content or read it as words, lines, bytes, or chunks as per your criteria. The extracted content will be split as a slice of the data separated by the delimiter set in the `Comma` attribute. ```go package main import ( "encoding/csv" "log" "os" ) func main() { f, err := os.Open("delimeter.txt") if err != nil { log.Fatal(err) } defer f.Close() reader := csv.NewReader(f) reader.Comma = ':' data, err := reader.ReadAll() if err != nil { log.Fatal(err) } log.Println(data) } ``` ``` $ cat delimiter.txt 10:22:2022 golang:21:read $ go run delimiter.go 2022/10/23 22:40:44 [[10 22 2022] [golang 21 read]] ``` In the above example, the delimiter is set as `:` so by using the `Comma` attribute we can set the delimiter. By using the `NewReader` function, we fetch the reader object and by using the `ReadAll` function associated to the reader object, we read the contents. The content is fetched as a slice of strings which will be separated by the delimiter. ### Reading File word by word We can even use [ScanWords](https://pkg.go.dev/bufio#ScanWords) to read a file word by word. A word can be a collection of characters that are separated by space. Instead of reading it line by line, this function reads the file content after a space. ```go package main import ( "bufio" "log" "os" ) func main() { f, err := os.Open("sample.txt") if err != nil { log.Fatal(err) } defer f.Close() scanner := bufio.NewScanner(f) scanner.Split(bufio.ScanWords) if err := scanner.Err(); err != nil { log.Fatal(err) } wordlist := []string{} for scanner.Scan() { word := scanner.Text() wordlist = append(wordlist, word) log.Println(word) } log.Println(wordlist) } ``` ``` $ go run word.go 2022/10/23 22:42:03 Golang 2022/10/23 22:42:03 is 2022/10/23 22:42:03 a 2022/10/23 22:42:03 programming 2022/10/23 22:42:03 language. 2022/10/23 22:42:03 created: 2022/10/23 22:42:03 2007 2022/10/23 22:42:03 type:static 2022/10/23 22:42:03 [Golang is a programming language. created: 2007 type:static] ``` With the `ScanWords` function, we can read the contents of the file word by word. The scanner object which has the actual content of the file is split by the `Split` function, the split criteria are used as Word, where the delimiter will be used as space. The `wordlist` is a slice of strings to which we append the string `word` that in turn is read from the `scanner.Text()` function. ### Reading Files in chunks We can even read files in chunks, a chunk is a collection/array of bytes. We can specify the number of bytes we want to read in one go and the file reader will scan the content as a slice of that number of bytes each iteration. The [Read](https://pkg.go.dev/bufio#Reader.Read) funciton takes in a slice of bytes and will return the number of bytes in the reader object. ```go package main import ( "bufio" "io" "log" "os" ) func main() { f, err := os.Open("sample.txt") if err != nil { log.Fatal(err) } defer f.Close() reader := bufio.NewReader(f) chunk_size := 16 chunk_list := []string{} buf := make([]byte, chunk_size) for { n, err := reader.Read(buf) if err != nil { if err != io.EOF { log.Fatal(err) } break } chunk_list = append(chunk_list, string(buf[0:n])) } for _, chunk := range chunk_list { log.Print(chunk) } } ``` ``` $ go run chunks.go 2022/10/23 22:44:41 Golang is a prog 2022/10/23 22:44:41 ramming language 2022/10/23 22:44:41 . created: 2007 2022/10/23 22:44:41 type:static ``` In the above example, we have opened the file and loaded the content into the `f` variable. the contents are read with the help of the `NewReader` function which returns a reader object which further can be used to read contents into chunks of bytes. The `chunk_size` defines the size we want to use for reading the content, `chunk_list` as a slice of strings which will hold the slice of chunks/bytes as a type caste into a slice of strings. With the `Read` function, the bytes are read into the function, and the buffer is split as per the chunk size obtained in the `Read` function. We append the slice of bytes into the sliced array and thereby we obtain the slice of strings. ### Read file character by character We can even read file each character at a time, using the [ScanRunes](https://pkg.go.dev/bufio#ScanRunes) function, this function scans a single rune/byte at a time. So, we can scan these runes one at a time and store them as a slice of bytes. Thereby we will have the content of the file stored as a slice of bytes. ```go package main import ( "bufio" "log" "os" ) func main() { f, err := os.Open("sample.txt") if err != nil { log.Fatal(err) } defer f.Close() scanner := bufio.NewScanner(f) scanner.Split(bufio.ScanRunes) if err := scanner.Err(); err != nil { log.Fatal(err) } charlist := []byte{} for scanner.Scan() { char := byte(scanner.Text()[0]) charlist = append(charlist, char) } log.Println(charlist) } ``` ``` $ go run char.go 2022/10/23 22:48:55 [71 111 108 97 110 103 32 105 115 32 97 32 112 114 111 103 114 97 109 109 105 110 103 32 108 97 110 103 117 97 103 101 46 10 99 114 101 97 116 101 100 58 32 50 48 48 55 10 116 121 112 101 58 115 116 97 116 105 99 10] ``` We can see in the above example, the output is a slice of byte, hence these are `uint8`, we can cast them to `string` and obtain the equivalent ASCII representation of the bytes. The `ScanRunes` function allows us to read the content from the reader object as a rune as we split the reader object into the unit bytes/runes. That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/mr-destructive/100-days-of-golang/tree/main/scripts/files/read/) GitHub repository. ## Conclusion In this section, we explored the functions and packages related to file reading. We saw how we can use packages like `os`, `bufio`, `encoding`, etc. to read files in a different way. We saw how to read files as a single string, line by line, word by word, character by character, in chunks, and also with a custom delimiter. Hopefully, the basics of file reading will have been cleared and with the examples, the syntactical construct was understood. Thank you for reading. If you have any queries, questions, or feedback, you can let me know in the discussion below or on my social handles. Happy Coding :) https://www.meetgor.com/posts/golang-paths Introduction In the 21st post of the series, we will be exploring the file paths in golang, we will be exploring how we can deal with paths. By using packages l Sun, 02 Oct 2022 00:00:00 UTC ## Introduction In the 21st post of the series, we will be exploring the file paths in golang, we will be exploring how we can deal with paths. By using packages like `os`, `path`, `io`, we can work with file systems and operating system-specific details. In this section, we will see how to resolve paths, details from paths, extract relative or absolute paths, iterate over file systems, etc. Starting from this post, it will follow a specific topic in the upcoming few posts which will be covering files and paths. We will be talking about dealing with paths and files in golang. This post is just about working with paths. ## Resolving and Parsing Path In golang, the [os](https://pkg.go.dev/os) and the [path](https://pkg.go.dev/path) packages are quite helpful in working with paths. We use the `path�ilpath` package specifically for working with paths and file structures. ### Get the current working directory To get the path for the current working directory, we can use the [os.Getwd()](https://pkg.go.dev/os#Getwd) function. The function returns a-ok, an error-like object if the working directory exists it will return the absolute path to the directory else if the path is deleted or corrupted while processing, it will give an error object. ```go package main import( "os" "log" ) func main() { dir, err := os.Getwd() if err != nil { log.Println(err) } else { log.Println(dir) } } ``` ``` $ pwd /home/meet/code/techstructive-blog $ go run main.go 2022/10/01 19:19:09 /home/meet/code/techstructive-blog ``` So, as we can see the `Getwd` the function returns an absolute path to the current working directory which will be the path from which you will be executing/running the script file. ### Get the path to the home directory We can even get the home directory path like the `/home` followed by the user name on Linux and the User Profile with the name for Windows. The [UserHomeDir()](https://pkg.go.dev/os#UserHomeDir), returns the home directory for the user from which the file is being executed. The return value is simply an string just like the `Getwd` function. ```go package main import( "os" "log" ) func main() { dir, err := os.UserHomeDir() if err != nil { log.Println(err) } else { log.Println(dir) } } ``` ``` $ echo $HOME /home/meet/ $ go run main.go 2022/10/01 19:35:50 /home/meet ``` So, as expected, the `UserHomeDir` function returns the path string to the home directory of the user. ### Get path from a file name string Let's say, we give in a filename and we want the absolute path of it. The [path/filepath](https://pkg.go.dev/path/filepath) package provides the [Abs](https://pkg.go.dev/path/filepath#Abs) function that does exactly that. The function returns a path string of the parameter parsed as a string to a directory or a file name. The function might as well return an error as the file path might not existing or the file might have got deleted, so we'll have to call the function with the ok, error syntax. ```go package main import( "path/filepath" "log" ) func main() { file_name := "default.md" log.Println(file_name) dir, err := filepath.Abs(file_name) if err != nil { log.Println(err) } else { log.Println(dir) } } ``` ``` $ go run main.go 2022/10/01 19:52:23 default.md 2022/10/01 19:52:23 /home/meet/code/techstructive-blog/default.md ``` As we can see the file `default.md` was parsed in the `Abs()` function and it returned the absolute path of the file. ### Get Parent Directory from a Path We can get the parent directory for a given path, if the path is to a file, we return the absolute path to the parent directory of the file, or if the path is to a folder, we return the folder's parent directory. ```go package main import( "path/filepath" "log" ) func main() { file_name := "drafts/default.md" //file_name := "drafts/" path, err := filepath.Abs(file_name) if err != nil { log.Println(err) } else { //log.Println(path) log.Println(filepath.Dir(path)) } } ``` ``` $ go run main.go 2022/10/01 19:58:45 /home/meet/code/techstructive-blog/drafts $ go run main.go 2022/10/01 19:58:45 /home/meet/code/techstructive-blog ``` As we can see when we parse in a file path i.e. `drafts/default.md`, the `Dir` the method returns a path to the parent folder, and even if we parse the directory path i.e. `drafts/`, the method returns the parent of that directory. ### Get the last file/folder for a given Absolute Path Golang also provides a way to get the file/directory name from a path string using the [Base](https://pkg.go.dev/path/filepath#Base) function provided in the [path/filepath](https://pkg.go.dev/path/filepath) package. ```go file_name := "default.md" dir, err := filepath.Abs(file_name) if err != nil { log.Println(err) } else { log.Println(dir) log.Println(filepath.Base(dir)) } ``` ``` $ go run main.go 2022/10/01 19:58:45 /home/meet/code/techstructive-blog/drafts/default.md 2022/10/01 20:19:28 default.md ``` So, the function `Base` will return the last element in the path, it can be a file or a directory, just returns the name before the last `\`. In the above example, we start with a filename `default.md` but set the dir as the absolute path to that file and again grab the file name using the `Base` function. ## Fetching details from a Path We can even use utility functions for dealing with paths in golang like for checking if a file or path exists, if a path is a file or a directory, grabbing file name and extension, etc. The `path/filepath` and the `os` the package helps with working with these kinds of operations. ### Check if a path exists We can use the [os.Stat](https://pkg.go.dev/os#Stat) function along with the [os.IsNotExist](https://pkg.go.dev/os#IsNotExist) for finding if a path is existing or not. The Stat function returns a [FileInfo](https://pkg.go.dev/io/fs#FileInfo) object or an error. The `FileInfo` object will have methods such as `Name()`, `IsDir()`, `Size()`, etc. If we get an error, inside the Stat method, the error will probably arise if the path does not exist, so inside the `os` package, we also have the `IsNotExist()` method, that returns a `boolean` value. The method returns `true` if the parsed error indicates that the path doesn't exist and `false` if it exists. ```go package main import( "path/filepath" "log" "os" ) func main() { file_name := "drafts/default.md" path, err := filepath.Abs(file_name) if err != nil { log.Println(err) } else { if _, err := os.Stat(path); os.IsNotExist(err) { log.Println("No, " + path + " does not exists") } else { log.Println("Yes, " + path + " exists") } } } ``` ``` $ go run main.go 2022/10/01 20:51:31 Yes, /home/meet/code/techstructive-blog/drafts/default.md exists ``` So, from the above example, the program will log if the path is present in the system or not. The error is parsed from the `Stat` method to the `IsNotExist` method for logging relevant messages. Since the directory exists, we get the path exists log. ### Check if a path is a file or directory The `FileInfo` object returned from the `Stat` the method provides a few methods such as `IsDir()` that can be used for detecting if a given path is a directory or not. The function simply returns a `boolean` value if the provided path points to a directory or not. Since we have to parse the path to the `IsDir()` function, we convert the file string into a path using the `Abs` method and then check if the path actually exist with the `Stat()` method. ```go package main import( "path/filepath" "log" "os" ) func main() { file_name := "drafts/default.md" //file_name := "drafts/" path, err := filepath.Abs(file_name) if err != nil { log.Println(err) } else { if t, err := os.Stat(path); os.IsNotExist(err) { log.Fatal("No, " + path + " does not exists") } else { log.Println(path) log.Println(t.IsDir()) } } } ``` ``` $ go run main.go 2022/10/01 20:55:20 /home/meet/code/techstructive-blog/drafts/default.md 2022/10/01 20:55:20 false $ go run main.go 2022/10/01 20:55:20 /home/meet/code/techstructive-blog/drafts/ 2022/10/01 20:55:20 true ``` So, by running the program for a file and a directory, we can see it returns `true` if the path is a directory and `false` if the provided path is a file. In the above example, since the `drafts/defaults.md` is a file, it returned `false`, and for the next example, when we set the path `drafts/` it returns `true` as the path provided is a directory. ### Get File Extension from path By using the [path](https://pkg.go.dev/path) package, the extension of a given path can be fetched. The [Ext](https://pkg.go.dev/path#Ext) method can be used for getting the extension of the provided path string, it doesn't matter if the provided path is exists or not, is absolute or relative, it just returns the text after the last . in the string. But if we are working with real systems it is good practice to check if the file or path actually exists. ```go package main import( "path/filepath" "log" "path" ) func main() { file_name := "default.md" dir, err := filepath.Abs(file_name) if err != nil { log.Println(err) } else { file_ext := path.Ext(dir) log.Println(file_ext) } } ``` ``` $ go run main.go 2022/10/01 21:03:23 .md ``` The above example demonstrates how we can get the extension of a file using the `Ext()` method in the `path` package. Given the string path as `default.md`, the function returned `.md` which is indeed the extension of the provided file. ### Get Filename from path We can even get the file name from a path in golang using the [TrimSuffix](https://pkg.go.dev/strings#TrimSuffix) method in the [strings](https://pkg.go.dev/strings) package. The `TrimSuffix` method trim the string from the provided suffix, like if we have a string `helloworld` and we provide the suffix as `world`, the `TrimSuffix` the method will return the string `hello`, it will remove the suffix string from the end of the string. ```go package main import( "path/filepath" "log" "path" "strings" ) func main() { file_name := "default.md" dir, err := filepath.Abs(file_name) if err != nil { log.Println(err) } else { file_ext := path.Ext(dir) log.Println(file_ext) log.Println(strings.TrimSuffix(dir, file_ext)) log.Println(strings.TrimSuffix(file_name, file_ext)) //log.Println(strings.TrimSuffix(dir, path.Ext(dir))) //log.Println(strings.TrimSuffix(file_name, path.Ext(dir))) } } ``` ``` $ go run main.go 2022/10/01 21:09:39 .md 2022/10/01 21:09:39 /home/meet/code/techstructive-blog/default 2022/10/01 21:09:39 default ``` We can use the `TrimSuffix` method to remove the extension as the suffix and it returns the path which we get as the file name. The `TrimSuffix` method returns the path after removing the extension from the path. ## List Files and Directories in Path In golang, we can use the `io` and the `path/filepath` packages to iterate over the file paths. Suppose, we want to list out all the files or directories in a given path, we can use certain functions such as `Walk`, `WalkDir` to iterate over a path string. There are certain types of iterations we can perform based on the constraints we might have, like iterating over only files, or directories, not including nested directories, etc. We'll explore the basic iterations and explain how we fine-tune the iteration based on the constraints. ## List only the files in the Path The first example, we can take is to simply list out only the files in the current path directory, we don't want to list out the file in nested directories. So, it will be like a simple ls command in Linux. Let's see how we can list out the files in the given path. We can even use `path/filepath` package to iterate over a given path and list out the directories and files in it. The [filepath.Walk](https://pkg.go.dev/path/filepath#Walk) or the [WalkDir](https://pkg.go.dev/path/filepath#WalkDir) method is quite useful for this kind of operation, the function takes in a path string and a [WalkFunc](https://pkg.go.dev/path/filepath#WalkFunc) or the [WalkDirFunc](https://pkg.go.dev/io/fs#WalkDirFunc) Function, the walk function are simply used for walking of a path string. Both functions take two parameters, the first being the string which will be the file system path where we want to iterate or walk, and the next parameter is the function either [WalkFunc](https://pkg.go.dev/path/filepath@go1.19.1#WalkFunc) or [WalkDirFun](https://pkg.go.dev/io/fs#WalkDirFunc) respectively. Both functions are similar but a subtle difference in the type of parameter both take in. ### WalkDir Function The `WalkDir` function takes in the parameters such as a `string` of the file path, the [fs.DirEntry](https://pkg.go.dev/io/fs#DirEntry) object and the `error` if any. The function returns an `error` if there arises any. We have to call the function with the parameters of a string and a function object which will be of type `type WalkDirFunc func(path string, d DirEntry, err error) error`. We can even use Walk the function to iterate over the given path. ### Walk Function The `Walk` function takes in the parameters such as a `string` of the file path, the [fs.FileInfo](https://pkg.go.dev/io/fs#FileInfo) object and the `error` if any. The function returns an `error` if there arises any. We have to call the function with the parameters of a string and a function object which will be of type `type WalkFunc func(path string, info fs.FileInfo, err error) error`. It might be a user preference to select one of the functions for iterating over the file system, but the [documentation](https://pkg.go.dev/path/filepath#Walk) says, the `Walk` function is a little bit inefficient compared to the `WalkDir` function. But if performance is not an issue, you can use either of those based on which type of file system object you are currently working with. ```go package main import( "path/filepath" "log" "io/fs" ) func main() { var files []string dir_path := "." err := filepath.WalkDir(dir_path, func(path string, info fs.DirEntry, err error) error { dir_name := filepath.Base(dir_path) if info.IsDir() == true && info.Name() != dir_name{ return filepath.SkipDir } else { files = append(files, path) return nil } }) if err != nil { panic(err) } for _, file:= range files { log.Println(file) } } ``` ``` $ go run walk.go 2022/10/02 12:07:17 . 2022/10/02 12:07:17 .dockerignore 2022/10/02 12:07:17 .gitignore 2022/10/02 12:07:17 CNAME 2022/10/02 12:07:17 Dockerfile 2022/10/02 12:07:17 README.md 2022/10/02 12:07:17 markata.toml 2022/10/02 12:07:17 requirements.txt 2022/10/02 12:07:17 textual.log ``` In the above example, we have used the `WalkDir` method for iterating over the file system, the directory is set as `.` indicating the current directory. We parse the first paramter as the string to the `WalkDir` function, the next parameter is a function so we can either create it separately or just define an `anonymous function`. It becomes a lot easier to write an `anonymous function` rather than writing the function separately. So, we have created the `dir_name` variable which parses the `dir_path` from the parameter to the function and gets the name of the directory or file. We can then fine-tune the requirements of the iteration of the directory, i.e. make checks if the path is a file or a directory and if we want to exclude any specific files with certain extensions or directories with a certain name, etc. In this example, we have added a check if the path is a directory(using `info.IsDir()`) and if the directory name is not the same as the parsed path(i.e. exclude the nested directories) we skip these types of directories (using [filepath.SkipDir](https://pkg.go.dev/io/fs#SkipDir)). So we only look for the files in the current directory or the directory which we provided in the paramter as `dir_path`. We append those paths into the files array using the `append` method. Finally, we check for errors in the parsed parameter while iterating over the file system and `panic` out of the function. We can then simply iterate over the files slice and print or perform operations as required. ### All the files in the Path (inside directories) We can also list all the files within the folders provided in the path string by removing the directory name check. We will only append the file type to the file slice rather than appending all the directories. ```go package main import( "path/filepath" "log" "io/fs" ) func main() { var files []string root := "static/" err := filepath.WalkDir(root, func(path string, info fs.DirEntry, err error) error { if info.IsDir() { return nil } else { files = append(files, path) return nil } }) if err != nil { panic(err) } for _, file:= range files { log.Println(file) } } ``` ``` $ go run walk.go 2022/10/02 12:08:22 static/404.html 2022/10/02 12:08:22 static/CNAME 2022/10/02 12:08:22 static/index.html 2022/10/02 12:08:22 static/main.css 2022/10/02 12:08:22 static/projects/index.html 2022/10/02 12:08:22 static/social-icons.svg 2022/10/02 12:08:22 static/tbicon.png ``` As we can see the iteration resulted in printing all the files in the given path including the files in the subdirectories. The static directory had the projects directory as a subdirectory in the path, hence we are listing the files in that directory as well. ### Recursive directories in the Path We can also append the directory names as well as file names by completely removing the `info.IsDir()` check and add the printing out of the relevant information as dir and files depending on the type. We can also maintain different lists or slices for directory and file and append them accordingly. ```go package main import( "path/filepath" "log" "io/fs" func main() { var files []string root := "static/" err := filepath.WalkDir(root, func(path string, info fs.DirEntry, err error) error { files = append(files, path) var f string if info.IsDir() { f = "Directory" } else { f = "File" } log.Printf("%s Name: %s ", f, info.Name()) return nil }) if err != nil { panic(err) } for _, file:= range files { log.Println(file) } } ``` ``` $ go run walk.go 2022/10/02 12:09:48 Directory Name: static 2022/10/02 12:09:48 File Name: 404.html 2022/10/02 12:09:48 File Name: main.css 2022/10/02 12:09:48 Directory Name: projects 2022/10/02 12:09:48 File Name: index.html 2022/10/02 12:09:48 File Name: social-icons.svg 2022/10/02 12:09:48 File Name: tbicon.png 2022/10/02 12:09:48 static/ 2022/10/02 12:09:48 static/404.html 2022/10/02 12:09:48 static/index.html 2022/10/02 12:09:48 static/main.css 2022/10/02 12:09:48 static/projects 2022/10/02 12:09:48 static/projects/index.html 2022/10/02 12:09:48 static/social-icons.svg 2022/10/02 12:09:48 static/tbicon.png ``` We can see that the directories and files getting logged which are present in the given path. In the output above, the projects the directory is getting walked along with the files present inside the directory. This is how we can use the Walk method to iterate over directories in a file system. ### All the folders in the Path (only directories) If we want to print only the directories, we can again add checks in the funciton body, we can simply append the path name when the path returns `true` on `IsDir` function call. ```go package main import( "path/filepath" "log" "io/fs" ) func main() { var folders []string root := "static/" err := filepath.WalkDir(root, func(path string, info fs.DirEntry, err error) error { dir_name := filepath.Base(root) if info.IsDir() { folders = append(folders, info.Name()) return nil } else if info.IsDir() && dir_name != info.Name() { return filepath.SkipDir } return nil }) if err != nil { panic(err) } for _, folder := range folders{ log.Println(folder) } } ``` ``` $ go run walk.go 2022/10/02 12:13:25 static 2022/10/02 12:13:25 projects ``` Here, we can see it lists all the folder names present in the given path, it will log all the nested directories as well. In the above example, the `static/` path in the local system had a projects directory and hence it prints the same, but that can be till the final depth of the file system. For all the examples on the `Walk` functions, you can check out the links on the GitHub repository: - [Walk and WalkDir function examples](https://github.com/Mr-Destructive/100-days-of-golang/blob/main/scripts/paths/walk.go) - [Walk and WalkDir function as anonymous function](https://github.com/Mr-Destructive/100-days-of-golang/blob/main/scripts/paths/walk_anonymous.go) ## Relative or Absolute Paths We have been using absolute paths in the above examples, but while navigating from one directory to other, we heavily make use of relative paths as they make it easier to move around. ### Check if a path is Absolute We can check if a path is absolute using the [IsAbs](https://pkg.go.dev/path#IsAbs) function, the function takes in a path string as a parameter and returns a boolean value. It returns `true` if the provided path is absolute else it returns `false`. ### Check if a path is Absolute ```go package main import ( "log" "os" "path/filepath" ) func main() { dir, err := os.Getwd() if err != nil { panic(err) } log.Println(dir) log.Println(filepath.IsAbs(dir)) dir = "../math" log.Println(dir) log.Println(filepath.IsAbs(dir)) } ``` ``` $ go run rel_abs.go 2022/10/02 14:38:44 /home/meet/code/techstructive-blog 2022/10/02 14:38:44 true 2022/10/02 14:38:44 ../math 2022/10/02 14:38:44 false ``` In the above example, we can see that when we parse `../math` indicating there's a `/math` directory, before the current directory(parent directory) we get `false`. But when we parse the path obtained from `Getwd()` function call or a path which is located from the root path will get the return value as `true`. ### Get the relative path from base to target path Let's say we are in a certain directory `/a/b/c/`, we want to move into `/a/c/d/`, we will have to move back two times and then move into `c` followed by the `d` directory. The relative path from `/a/b/c/` to `/a/c/d/` can be described as `../../c/d/`. We have a function in golang that does the same, basically creating a relative path from the base directory path to a target path. The function is provided in the path/filepath package as [Rel](https://pkg.go.dev/path/filepath@go1.19.1#Rel), the function takes in two parameters, both as a string representing paths. The first is the base path(like you are in) and the second is the target path (as the target to reach). The function returns the string representation of the absolute path from the base to the target directory. ```go package main import ( "log" "os" "path/filepath" ) func main() { dir, err := os.Getwd() if err != nil { panic(err) } dir, err = filepath.Abs("plugins/") s, err := filepath.Abs("static/projects/") if err != nil { log.Println(err) } log.Println(s) log.Println(dir) log.Println(filepath.Rel(s, dir)) } ``` ``` $ go run rel_abs.go 2022/10/02 12:26:09 /home/meet/code/techstructive-blog/static/projects 2022/10/02 12:26:09 /home/meet/code/techstructive-blog/plugins 2022/10/02 12:26:09 ../../plugins <nil> ``` We can see that the relative path from the two directories is given as the return string from the Rel function. ### Join paths The [Join](https://pkg.go.dev/path/filepath@go1.19.1#Join) method provided in the `filepath` package, is used for combining `n` number of path strings as one path. It separates the file paths with the operating system-specific separator like `/` for Linux and `\` for windows. ```go package main import ( "log" "path/filepath" ) func main() { dir, err := filepath.Abs("operators/arithmetic/") if err != nil { log.Println(err) } log.Println(filepath.Join("golang", "files")) log.Println(filepath.Join(dir, "/files", "//read")) } ``` ``` $ go run rel_abs.go 2022/10/02 12:30:37 golang/files 2022/10/02 12:30:37 /home/meet/code/techstructive-blog/operators/arithmetic/files/read ``` In the above example, we can see that it parses the path accurately and ignore any extra separators in the string path. That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/mr-destructive/100-days-of-golang/tree/main/scripts/paths) GitHub repository. ## Conclusion So, from the following post, we were able to explore the path package along with a few functions io as well as os package. By using various methods and type objects, we were able to perform operations and work with the file paths. By using functions to iterate over file systems, checking for absolute paths, checking for the existence of paths, etc, the fundamentals of path handling in golang were explored. Thank you for reading, if you have any queries, feedback, or questions, you drop them below on the blog as a [github discussion](https://www.meetgor.com/golang-paths/#comments), or you can ping me on my social handles as well. Happy Coding :) https://www.meetgor.com/posts/golang-error-handling Introduction Error handling is quite an important feature of any programming language to improve the quality and transparency between the user and the applicati Sat, 17 Sep 2022 00:00:00 UTC ## Introduction Error handling is quite an important feature of any programming language to improve the quality and transparency between the user and the application. By raising appropriate error messages, the user can get a clear idea about the things happening in the interface as well as the application can handle the errors with appropriate actions. In the 20th post of the series, we will be exploring the concept of error handling in golang. From this article, we will be able to learn the fundamentals of error or exception handling in golang, create custom error classes, raise and ignore error messages, and exit or redirect the flow of state of the application when an error is raised. ## Why we need Error Type We need [error handling](https://go.dev/blog/error-handling-and-go) and catching in order to stop or divert the flow of the application which will restrict the dubious or unintentional execution of the code. Let's say, for example, we have a string as an input, and the user skipped the input and the string is returned as empty, we don't want to execute the further program as the execution might depend on the value of the string. So, in order to catch these kinds of events, we might use errors and log the results for better transparency. ```go package main import ( "fmt" ) func main() { var s string n, err := fmt.Scanf("%s", &s) if err != nil { fmt.Println(err) // panic(err) // OR // return } else { fmt.Println(n) if s[0] == 'a' { fmt.Println(s) } } } ``` ``` $ go run main.go asdf 1 asdf $ go run main.go unexpected newline $ go run main.go wsd 1 ``` In the above example, we have a simple string `s` input, we will input the string using the `Scanf` function that will return an integer as the number of variables it has scanned and error if any. Here, as the function might return two values, we need two variables to call the function. The `n` variable stores the number of variables successfully scanned and the `err` as the err from the function generated. If there is an error, that is the value stored in `err` is not empty, we will log the error. And move into the rest of the program. This might be looking cool, but it doesn't break out of the program if there is an error. We want it to log the error as well as exit from the program. We can do that using the panic function. ## Catching Errors We'll see a few examples, where we will catch errors in some of the regularly used functions. These error messages can be used in deciding the next procedure to be run. ### Comma OK/Error syntax We use the [comma ok, error](https://go.dev/doc/effective_go#:~:text=is%20called%20the%20%E2%80%9C-,comma%20ok,-%E2%80%9D%20idiom.%20In%20this) syntax where we want multiple return values from a function. It is a narrowed syntax for a two-type return value function, we either return a value that we are expecting as `OK`, or we get an `error` from the function call. ```go ok, err := function() if err != nil { // handle error panic(err) } else { // work with the ok object fmt.Println(ok) } ``` In the above code, we have used the comma-ok, error syntax, the function call will return two objects either an expected object or an error object if there were errors in the processing. We handle the error if the error object is not empty i.e. it contains something, else we can do the rest of the required processing of the program. We can even ignore the `err` or the `ok` object using the `_` i.e. to say a don't care variable. Remember you can ignore either of the values and not both. It is not recommended to ignore errors but if you know the obvious thing to process, you might as well sometimes. A more compressed code might look something like below: ```go if ok, err := function(); err != nil { // handle error panic(err) } else { // work with the ok object fmt.Println(ok) } ``` The above code wraps the initialization of `ok, err` objects by calling the function inside the if statement and checking for the error. ### Making HTTP requests Let's say we have a URL, and we want to check if the site exists or not, we can run the [http.Get](https://pkg.go.dev/net/http@go1.19.1#Client.Get) function from the [net/http](https://pkg.go.dev/net/http) package. We will parse a URL in the function and this function also returns a [Response](https://pkg.go.dev/net/http#Response) type object and an error object if there are any errors generated during the function call. If there is an error while we call the function, we simply log the error and panic out of the program. Else we can log the status code and do further processing. ```go package main import ( "fmt" "net/http" ) func main() { url := "https://meetgor.com/" resp, err := http.Get(url) if err != nil { fmt.Println(err) panic(err) } else { fmt.Println(resp.StatusCode) } } ``` ``` $ go run web.go URL: https://meetgor.com/ 200 $ go run web.go URL: htts://meetgor.com/ Get "htts://meetgor.com/": unsupported protocol scheme "htts" panic: Get "htts://meetgor.com/": unsupported protocol scheme "htts" goroutine 1 [running]: main.main() /home/meet/code/100-days-of-golang/scripts/errors/https.go:14 +0x170 exit status 2 ``` This is how we can validate a URL handling the error if the parsed URL is invalid or does not exist. There is one more variation of the above code style, it is a bit compressed and might be just a syntactic change. ```go package main import ( "fmt" "net/http" ) func main() { url := "https://meetgor.com/" if resp, err := http.Get(url); err != nil { fmt.Println(err) panic(err) } else { fmt.Println(resp.StatusCode) } } ``` This can be used wherever you are using the `ok, err` kind of syntax, but I prefer the clean syntax so I won't move ahead with this. ### Opening or Handling of File We can even use error handling while dealing with Files or Folders. We can use the [os](https://pkg.go.dev/os) package to read the file in golang. The [Open](https://pkg.go.dev/os#Open) function will read the file if it exists or else it will return an error. We can catch the error from the comma `ok,error` syntax and do the required processing in the program. ```go package main import ( "fmt" "os" ) func main() { file_name := "hello.txt" file, err := os.Open(file_name) if err != nil { fmt.Println("Error: ", err) // Create a File // _, err := os.Create(file_name) // if err != nil { // fmt.Println(err) // } // fmt.Println("Created File", file_name) // file, _ = os.Open(file_name) } fmt.Println(file.Stat()) } ```  We can use the error as a hint that the file doesn't exist and create a file and then move toward the actual execution of the program. We can also ignore the file object while creating the file, as we are interested in only knowing that the file is just created without any errors, we use the `_` to ignore the variable in the assignment in the function call. Inside the commented code, we use the [Create](https://pkg.go.dev/os#Create) function to create a file and check for any errors in the process. We finally create the file and `Open` the newly created file. ## Custom Error We can create custom error types in golang with the help of interfaces and structs. An Error struct will simply consist of a string `message`, that string will display the error. By overriding or creating the `Error` method as an interface for the struct we can construct custom errors. ```go type Invalid_URL_Error struct { message string } func (e Invalid_URL_Error) Error() string { return "Invalid URL" } ``` Here, we have the `Invalid_URL_Error` as the custom struct name and the `Error()` method as an interface that will print the error log. This Error method will be used while raising errors in the program. It might be called from another function while doing the actual processing of the URL while sending a GET request. Further, we can call this custom error method when we wish, by using the package functions, we can override the function call with the custom method. ```go package main import ( "fmt" "net/http" ) type Invalid_URL_Error struct { message string } func (e Invalid_URL_Error) Error() string { return "Invalid URL" } func main() { url := "htt://meetgor.com" response, err := http.Get(url) if err != nil { fmt.Println(Invalid_URL_Error{}) fmt.Println(err) } else { fmt.Println(response) defer response.Body.Close() } } ``` ``` $ go run custom_error.go Invalid URL Get "htt://meetgor.com": unsupported protocol scheme "htt" ``` In the above code, we are basically calling the function `http.Get` that will return a `Response` or an `err` object. We can even call the custom error method with an empty `Invalid_URL_Error` object, this will call the function `Error` from that interface. The function will print the custom error message and thereby we are able to log the custom error message for the invalid URL example. Also, we can parse the default error method to the custom error method which will get us additional information inside the error interface method. ```go package main import ( "fmt" "net/http" ) type Invalid_URL_Error struct { message string } func (e Invalid_URL_Error) Error() string { return "Invalid URL : " + e.message } func main() { url := "htt://meetgor.com" response, err := http.Get(url) if err != nil { fmt.Println(Invalid_URL_Error{err.Error()}) } else { fmt.Println(response) defer response.Body.Close() } } ``` ``` $ go run custom_error.go Invalid URL : Get "htt://meetgor.com": unsupported protocol scheme "htt" ``` When the URL is invalid, we will call the custom error interface by parsing the default `err.Error` method. This will get us the error object from the main method to our custom interface. That is how we will be able to fetch additional information about the error from the interface with the `.` operator as `e.message`. So, the syntax is `Invalid_URL_Error{err.Error()}`, i.e. an object of type `Invalid_URL_Error` with the message set as the value returned from the default `Error()` function. hence we can implement the custom error message. We also need to look for the response object and close the Response Body as it is mandatory to do so and the responsibility of the caller. ### Creating a function that returns two values (ok,error) We can even nest the calling of this error method inside another function. This will give us a good overview of how to deal with errors more thoroughly. We will construct a function with two return values one can be any normal desirable object (which we want from the function call) and the other as an error. This will check for any cases that we can call the custom error and return that error interface and the object which was to be returned will be nil if there is an error. If there are no errors, we will return the object and set the error as nil. This way, we can use the `ok, error` syntax while calling this function and thereby make it a lot easier in case of complex programs or multiple types of errors. ```go package main import ( "fmt" "net/http" ) type Invalid_URL_Error struct { message string } func (e Invalid_URL_Error) Error() string { return "Invalid URL" } func get_resp(url_link string) (http.Response, error) { resp, err := http.Get(url_link) if err != nil { return http.Response{}, &Invalid_URL_Error{} } else { defer resp.Body.Close() return *resp, nil } } func main() { url := "htts://meetgor.com" resp, err := get_resp(url) if err != nil { fmt.Println(err) } else { fmt.Println(resp) } } ``` ``` $ go run errors.go Invalid URL ``` This is the simple function `get_resp` which will either fetch the `Response` or `error`. We call the `http.Get` method internally in the function and if something is invalid, it will return an empty Response but the error will be an `Invalid_URL_Error` object which is the custom error class. This means, that if we have an error, we will return a string object from the `Error` method in the interface and if there is no error, we will return the `Response` object and the error will be set as `nil`. Hence, you can now realize, why we check for `err != nil`, it is used for checking if an error object has returned a string or not. As said earlier, we also need to close the request Body. Further, we can pass the default error method to the custom error class as `Invalid_URL_Error{err.Error()}`. This will ensure, we get additional context from the custom error interface. ```go package main import ( "fmt" "net/http" ) type Invalid_URL_Error struct { message string } func (e Invalid_URL_Error) Error() string { return "Invalid URL : " + e.message } func get_resp(url_link string) (http.Response, error) { resp, err := http.Get(url_link) if err != nil { return http.Response{}, &Invalid_URL_Error{err.Error()} } else { defer resp.Body.Close() return *resp, nil } } func main() { url := "htts://meetgor.com" resp, err := get_resp(url) if err != nil { fmt.Println(err) } else { fmt.Println(resp) } } ``` ``` $ go run custom_error Invalid URL : Get "htts://meetgor.com": unsupported protocol scheme "htts" ``` That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/mr-destructive/100-days-of-golang/tree/main/scripts/error-handling/main.go) GitHub repository. ## Conclusion From this article, we were able to understand the basics of error handling in golang. We can now work with handling errors in function calls and create custom error interfaces. Thank you for reading, if you have any queries, feedback, or questions, you can freely ask me on my social handles. Happy Coding :) https://www.meetgor.com/posts/golang-interfaces Introduction In the 19th post of the series, we will be taking a look into interfaces in golang. Interfaces allow us to create function signatures common to dif Sat, 10 Sep 2022 00:00:00 UTC ## Introduction In the 19th post of the series, we will be taking a look into interfaces in golang. Interfaces allow us to create function signatures common to different structs or types. So, we can allow multiple structs to have a common interface(method) that can have different implementations. ## What are Interfaces Interface as the name suggests is a way to create methods that are common to different structures or types but can have different implementations. It's an interface to define the method or function signatures but not the implementation. Let's take an example of `Laptop` and `Phone` having the functionality of wifi. We can connect to wifi more or the less in a similar way on both devices. The implementation behind the functionality might be different but they share the same operation. The WiFi can act as an interface for both devices to connect to the internet. ## Define an Interface To declare an interface in golang, we can use the `interface` keyword in golang. An interface is created with the type keyword, providing the name of the interface and defining the function declaration. Inside the interface which acts as a struct of general method signatures. The method signatures usually consist of the name of the function with its parameters if any and the return type of the function. ```go package main import "fmt" type Player struct { name string health int } type Mob struct { name string health int is_passive bool } type Creature interface { intro() string //attack() int //heal() int } func main() { player := Player{name: "Steve"} mob := Mob{name: "Zombie"} fmt.Println(player) fmt.Println(mob) } ``` ``` go run main.go {Steve 0} {Zombie 0 false} ``` In this above example, we have created an interface called `Creature`. There are a few structs that we have to define like `Player` and `Mob`, these two methods have a few attributes like `name` as `string` and `health` as `int` which are common in both structs but the `Mob` struct has an additional attribute `is_passive` as a `boolean` value. The `Creature` is an interface that will define certain function signatures, here we have declared `intro`, `attack`, and `heal` as the methods bound to the Creature interface. This means, that any object which satisfies the Creature interface, can define the methods associated with the interface. ## Defining Interfaces Once we have declared the interface method signatures, we can move into defining the functionality of these methods depending on the struct. If we want a different working method for different types of struct objects passed we can define those for each type of struct that we want. Here, we have two types of structs namely `Creature` and `Mob`, based on the struct we can define the `intro` method for them individually. ```go package main import "fmt" type Creature interface { intro() string attack(*int) int } type Player struct { name string health int } type Mob struct { name string health int category bool } func (p Player) intro() string { fmt.Println("Player has spawned") return p.name } func (p Player) attack(m_health *int) int { fmt.Println("Player has attacked!") *m_health = *m_health - 50 return *m_health } func (m Mob) intro() string { fmt.Printf("A wild %s has appeared! ", m.name) return m.name } func (m Mob) attack(p_health *int) int { fmt.Printf("%s has attacked you! -%d ", m.name, 30) *p_health = *p_health - 30 return *p_health } func main() { player := Player{name: "Steve", health: 100} mob := Mob{name: "Zombie", health: 140} fmt.Println(player.intro()) fmt.Println(mob.intro()) fmt.Println(mob) fmt.Println(player) fmt.Println(player.attack(&mob.health)) fmt.Println(mob.attack(&player.health)) fmt.Println(mob) fmt.Println(player) } ``` ``` go run main.go Player has spawned Steve A wild Zombie has appeared! Zombie {Zombie 140 false} {Steve 100} Player has attacked! 90 Zombie has attacked you! -30 70 {Zombie 90 false} {Steve 70} ``` As we can see, the method `intro()` is bound to both the struct depending on what struct signature is associated with the method. The method `intro` takes in the object struct associated as per the call and returns `string` as defined in the method signature. The `attack` method in the `Creature` interface is also implemented separately for the two structs. For the `Player` method, we simply take in a pointer to an integer and return an `int`. The parameter is the pointer to the mob health, and it returns the modified health. We take in a pointer to the mob or player health so as to parse in the actual value and not the copy of the value. If we modify the value, we want to reflect those changes in the actual object. So that is how we can use interfaces to construct dynamic operations on objects as well as different types of structs. We have seen a simple example of how to declare and define interfaces for given type structs. Also, we can pass by value as well as by pointers so as to define the behavior of the method whether to dynamically modify or change the values of the object associated with it. ## Examples of Interfaces There are quite some use cases of interfaces, in object-oriented programming, the above example fits the polymorphism feature quite well. The ability to reuse certain method signatures and define the functions as per requirement brings flexibility to the code structure. We will see a few examples for understanding how we can use interfaces in various ways. ### Type Switch Interface We can use an empty interface to check for the type of variable we have parsed. Using this empty interface we can create a kind of dynamic parameter to a function. ```go package main import ( "fmt" "strconv" ) func parse_int(n interface{}) int { switch n.(type) { case int: return (n).(int) * (n).(int) case string: s, _ := strconv.Atoi(n.(string)) return s case float64: return int(n.(float64)) default: return n.(int) } } func main() { num := parse_int(4) fmt.Println(num) num = parse_int("4") fmt.Println(num) num = parse_int(4.1243) fmt.Println(num) } ``` ``` go run main.go 16 4 4 ``` Here, we can see we have an interface as a parameter to the function `parse_int`, the return type is `int`, so the incoming parameter can be any valid type. But if we don't convert the given type into an appropriate int, it will result in an error as we are returning the int value of the parsed parameter. We are taking the parameter as `interface{}` which is an empty interface, this will contain the parameter parsed as an interface type. That's why we need to convert the interface object into an int or the parsed type of the interface. ### Interface Slice We can even create a slice of interfaces, which means we can initialize or group together various objects of different structs in a single slice. This might be helpful for calling functions associated with different objects via the interface very easily and in a much cleaner way. ```go package main import "fmt" type Creature interface { intro() string } type Player struct { name string health int } type Mob struct { name string health int category bool } func (p Player) intro() string { fmt.Println("Player has spawned") return p.name } func (m Mob) intro() string { var name string if m.name != "" { name = m.name } else { name = "Mob" } fmt.Printf("A wild %s has appeared! ", name) return m.name } func main() { entity := []Creature{Player{}, Mob{}, Mob{}, Player{}} for _, obj := range entity { fmt.Println(obj.intro()) } } ``` ``` Player has spawned A wild Zombie has appeared! A wild Zombie has appeared! Player has spawned ``` In the above example, we can see that the entity variable is created as a slice of interfaces `Creature`, i.e. various objects associated with the `Creature` interface can be contained in a single slice. There are 2 instances of Player and Mob each in the slice. We can further iterate over the slice as a range-based loop and thereby the functions associated with the interfaces can be called. Here, we have called the `intro` function. So, there are a lot of things that can be done with interfaces, we can create multiple interfaces for a single object struct and nest interfaces. Based on the use case of the program, interfaces can be used to reduce the boilerplate code as well as improve the readability of the code. That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/mr-destructive/100-days-of-golang/tree/main/scripts/interfaces/main.go) GitHub repository. ## Conclusion From this part of the series, we were able to understand the basics of interfaces using a few examples. We explored how interfaces can be used to bring in polymorphism in golang, also we can improve the readability of the code. The boilerplate code can be considerably reduced by using interfaces when dealing with structs and types. Hopefully, you found this post helpful and understood even the basics of interfaces in golang. Thank you for reading, if you have any queries, questions, or feedback, you can ping me on my social handles or in the comments. Happy Coding :) https://www.meetgor.com/posts/golang-closures Introduction In the previous part of the series, we covered and in this section, we will look into which are quite a cool concept for various things. Closures a Thu, 11 Aug 2022 00:00:00 UTC ## Introduction In the previous part of the series, we covered [anonymous functions](https://meetgor.com/golang-anonymous-functions/) and in this section, we will look into `closures` which are quite a cool concept for various things. Closures are basically a function that returns a function instead of a value, so basically we will leverage anonymous functions for creating closures. ## Simple Closures A simple closure can be constructed for understanding how we can use closures in golang. We will return a function from a function, that is a simple closure. So, in the below code example, we have created a function `gophy()` which takes no parameters but returns a function that returns a `string`. The function simply returns an anonymous function that returns a string. We will initialize the variable `g` that is assigned to the function `gophy` which will simply return a function call. We are not calling the function simply returning the call to the function `gophy` that has the return value as the anonymous function. We will simply have the function in the variable `g` rather than the simple value string. So we will have to call the `g` variable for actually returning the string. ```go package main import "fmt" func gophy() func() string{ return func() string{ return "Hello, Gophers!" } } func main() { // using clousure/anonymous function to return a value // that value can be assigned to the variable g := gophy() fmt.Println(g()) } ``` ``` $ go run simple.go Hello, Gophers! ``` So, that is how we can call the function `g` that will return a string, so we have the function body stored in the variable `g`. We can call it as many times as we want. ## Variable Scope in Closures We can even use variables that will remain in the function scope once it is initialized. So, let’s say we have a function that will increment the counter, but if we want to keep the counter the same throughout the program, we might have to use a global variable so as to maintain the context, but with closures, we will retain the value once we have initialized the function call. In the below example, we are creating the function `incrementer` that is a closure with int as the return type. We are initializing the variable `counter` that will be acting as the counter in the program, the function returns an anonymous function that will increment the counter and return it. Here, when we create an instance of the `increment` function it basically initializes the `counter` to `0` and returns the anonymous function as a call. Now, `c` will act as a function that has the counter variable bound to it and we can call `c` that will, in turn, call the anonymous function keeping the scope of the `counter` variable. So, each time we call the function `c` it will increment the counter and thus we keep the counter inside the scope of the function `incrementer` in the `c` variable. ```go package main import "fmt" func inrementer() func() int{ counter := 0 return func() int{ counter += 1 return counter } } func main() { c := inrementer() fmt.Println(c()) fmt.Println(c()) fmt.Println(c()) fmt.Println(c()) fmt.Println(c()) } ``` ``` $go run simple.go 1 2 3 4 5 ``` If we want to extend the functionality, we can even assign the function call `c()` to a variable and access the returned value which will be the current state of the counter. We can even use different scope or closures tied to a particular function, that is we can bind data to a different instances of a closure. ```go package main import "fmt" func inrementer() func() int{ counter := 0 return func() int{ counter += 1 return counter } } func main() { c1 := inrementer() fmt.Println(c1()) fmt.Println(c1()) fmt.Println(c1()) c2 := inrementer() fmt.Println(c2()) fmt.Println(c2()) fmt.Println(c2()) fmt.Println(c2()) } ``` ``` $go run simple.go 1 2 3 1 2 3 4 ``` Here we have `c1` and `c2` forming different closures and thereby we can have different scopes of the variables associated with it. The variable is bound to the instance it which was initialized, so we can see the different closure instances having different values. ## Factorial of a function with Closures We can create some interesting programs with closures, we will implement the calculation of factorial with closures in golang. This will be a `factorial` function that returns an anonymous function with the return type as `int`. The function will initialize the variable `fact` which will store the actual factorial value and `n` as the initial number for calculating the factorial of it. Inside the anonymous function, we will calculate the factorial and increment the number and simply return the factorial value from the function. The `fact` variable will contain the factorial of the number n, so here we can leverage the use of closures as we will maintain the state of the variable `fact` and `n` from the previous calls or the initialization of the function. Inside the `main` function, we have created the `f` variable and called the `factorial` function, so that will initialize the `fact` and `n` of the variable and thereby returning the anonymous function call. Now we can call the variable `f` as many times as we want that will simply return the factorial of the number incremented each time we call the function. ```go package main import "fmt" func factorial() func() int{ fact, n := 1, 1 return func() int{ fact = fact * n n += 1 return fact } } func main() { f := factorial() fmt.Println(f()) fmt.Println(f()) fmt.Println(f()) fmt.Println(f()) fmt.Println(f()) } ``` ``` $ go run simple.go 1 2 6 24 120 ``` So, we can see that the factorial is getting printed for each call and the number is being incremented at each call. So that's the basics of closures in golang, we can use closures to keep the content secured and encapsulated from different function calls. We can bind data with closures, with the help of anonymous functions a closure can be constructed and data can be bound to a particular function called scope. That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/mr-destructive/100-days-of-golang/tree/main/scripts/closures/main.go) GitHub repository. ## Conclusion From this post, we could understand the fundamentals of closures in golang. The basic concept of closures in golang was understood with a few examples. Thank you for reading, if you have any queries or feedback please leave them in the comments or on my social handles. Happy Coding :) https://www.meetgor.com/posts/pgcli-pg-from-terminal Introduction Have you ever used the Postgres database and did you know you don't have to launch PGAdmin every time you want to write SQL queries, you can write Sun, 07 Aug 2022 00:00:00 UTC ## Introduction Have you ever used the Postgres database and did you know you don't have to launch PGAdmin every time you want to write SQL queries, you can write down those queries even inside a terminal with a python package. It's PGCLI, that can act as an editor window for writing SQL queries for Postgres Databases. We can simply execute SQL queries in a PSQL-like shell with additional features like autocompletion, text-editing modes, file input/output options, etc. If you are like CLIs and love to play with backend systems like Postgres(database) then chances are you will love this article. This article will cover the basics of using the PGCLI tool to perform simply as well as advanced options in interacting with your Postgres database. ## PGCLI PGCLi is a python package that acts as a CLI for executing SQL queries in a Postgres database. PGCLI allows us to interact with any Postgres database via the command line, it can be a remote database or a local database, you can access it right away from your terminal. The package acts as a CLI for accessing and executing any SQL queries, so we can simply provide the options/parameters to plug the database in and access the schema and its related data. ## Installation There are a lot of ways pgcli can be installed, my preferred way is with `pipx`, it just works well and you don't have to install it every time or mess up your global python packages environment. ``` pipx install pgcli ```  We don't even require installing it when using pipx, as it will each time create a fresh instance of the pgcli in a virtual environment. We can now run the `pgcli` with parameters using the `pipx run` command. In the below command, we have used the `--help` options to list out all the commands and other options available with the `pgcli` command. ``` pipx run pgcli --help ```  If this command is working fine, it means we are ready to connect and interact with a database with pgcli. ## List local databases We can list all the databases in our local system by providing the hostname and the username, in my case, I have to provide the `localhost` and `postgres` as the hostname and the username respectively. Further, by providing the `--list` option, we can list down all the available databases on our system, basically for a given host and user. ``` pipx run pgcli --host localhost --user postgres --list OR pipx run pgcli -h localhost -U postgres -l ```  As, we can see, the command lists out all the available databases on the local server. This can help in getting an overview of the Postgres databases present in your local system. You need to enter the password for the username to access the database. ## Connect to a database To connect to an individual Postgres database we have two options, we can either use a URL string of postgres database or parse parameters to the command individually. It's easier to use the URL option because it wraps a lot of things in a single string rather than writing each parameters. ### Connect with the Postgres Connection URI We can simply pass the postgres connection URI for accessing a postgres database with pgcli. The URI has a specific pattern in the order where we need to specify the values like hostname, password, username, port, and database name. ``` postgresql://username:password@hostname:port/database_name ``` The above is the structure for a postgres connection URI, we parse in the `username`, `password` for that `username`, `hostname` with the `port`, and the `database name`. These parameters if parsed correctly can directly access the database. ``` pipx run pgcli postgresql://postgres:postgres@localhost:5432/techstructive_blog ```  This command will leave us in a prompt where we are basically in a PSQL shell in the provided database or host. For connecting to a local database, the hostname will likely be `localhost` and postgres database port is generally `5432` there might be exceptions to this. We can quit out of PGCLI with `CTRL + D` or `exit`. While using a remote database, the hostname and other details might be provided, for instance, we might have a Django application, in that project, we will have `database_name`, `port`, and details mentioned in the `DATABASES` settings, those fields can be checked and a postgres connection URI might be constructed. Let's take an example. ```python DATABASES = { "default": { "ENGINE": "django.db.backends.postgresql", "NAME": "blog", "USER": "meet", "PASSWORD": "postgres", "HOST": "localhost", } } ``` We have all the fields provided in the django project' settings file. We can use this to construct a postgres database connection URI. ``` pgcli postgresql://meet:postgres@localhost:5432/blog ``` So, that is how you can extract parameters from other technologies and frameworks for working with postgres URIs. ### Connect with CLI options/paramters We can parse the options like `hostname`, `password`, and `database name` manually with the CLI options that PGCLI provides. ``` pipx run pgcli --username postgres --host localhost --port 5432 --dbname techstructive_blog OR pipx run pgcli -u postgres -h localhost -p 5432 -d techstructive_blog ```  You will be prompted for a password for the database, after entering the password, you will be inside the psql shell. In there, we can execute SQL queries and other specific psql or pgcli commands. Once we have connected to a database, we can now try to work with the SQL queries and explore some of the commands and features provided by pgcli. ## List out all databases if connected to a server Let's say we don't know the database name, we are just connected to a database server, we can get the list of the databases inside the shell with the `\l+` command. This command is similar to the `--list` parameter in the PGCLI command, but the only difference is that we are executing it from the psql shell rather than our terminal shell. ``` # Let's say we connected to a database server pipx run pgcli --username postgres --host localhost --port 5432 # List out all databases in that server \l+ ```  So, that's how we would get the gist of all the databases inside the database server to which we are connected using the `\l+` command. ## List all tables in the database We can get a list of all the tables existing in the database we are logged in with the `\dt` command, we can simply enter the command in the prompt we are directed to once we are logged in to the postgres database. ``` \dt ```  We can use `\d <table_name>` to describe the details of the table provided. We get back the list of the attributes inside the table and the indexes of the relationships of the table as well as the referenced table details.  We also have other commands for displaying types, schemas, roles, views, etc. with the `\d` prefix, a detailed list can be found in the [PGCLI documentation](https://www.pgcli.com/commands). ## Executing queries We can execute SQL queries in the PGCLI prompt as normally we do in a PSQL shell. There are added benefits like certain modes of editing text and auto-completion. We'll execute a few queries and also try out other options for editing and saving queries. Normally for simple and easy tasks, we can write SQL queries right in the PSQL shell, it might be small and simple to construct a query. Let's take the simplest example to get all the attribute lists from a specific table in a database. ```SQL SELECT * FROM table_name ```  We get an excellent table display of output which we can view by scroll or arrow keys. We can even use `j` and `k` for moving vertically in an output table. ### Wriing modes in PGCLI We can use two modes in PGCLI prompt, one is the Emacs mode which is the default one and the other is Vi-mode for Vim fanboys. We can switch between these modes with `F4` key. The `Emacs-mode` is just a normal typing experience without any modes or shortcut macros and other stuff. Though it's nice that PGCLI offers a VI-mode, it is great for people using Vim, when using the Shell for editing a Query, this mode can be utilized.  We can use `hjkl` keybinding for movement and other normal mode keymaps like `w` for moving a word, or `b` for moving a word in the backward direction, `e` to the end of the word, etc. We can use `i` or `a` for entering into insert mode, `dd` to delete the entire line, `cc` or `S` to delete the current line and enter into insert mode, `yy` to yank line, and `p` to paste, `u` for undo, etc. We can even execute shell commands by using the `!` in normal mode where we will be prompted for the command executed in the same window. ### Autocompletion in PGCLI shell PGCLI shell has an auto-completion feature that will provide some suggestions based on the recent queries or commands executed or the default options. You can toggle the autocompletion with the `F2` key, it is officially called as `Smart-Completion` as you would see in the bottom left corner of the shell. ### Select an Editor to write queries We can use different editors for writing SQL queries. We can write a query from an editor by using the `�` command. The default editor is chosen which will be set in one of your `$EDITOR` or `$VISUAL` environment variables. You can set them manually in the bashrc file by exporting the value of `VISUAL` or `EDITOR` as the name of your editor. ```bash export VISUAL=vim OR export EDITOR=vim OR export VISUAL=gedit ``` Here, we can set `VISUAL` environment variable as the name of the editor or the path to it. The `VISUAL` environment variable is preferred first. For the details, we can check out the [editor command documentation](https://www.pgcli.com/editor). Once, we have configured the editor, we will be inside the editor after we enter the `�` command. We can write queries inside the editor and quit after we are done, once the editor is closed, the text is read from the editor and entered in the psql shell. This will make us wait for the enter command for executing the query. We can even execute more than one query at a time if we use the `;` at the end of each query. Also, we can press `v` in VI-normal mode, to open the current query in the default editor. ### Save the query output to a file Let's say we have written a query inside the psql shell and we want the output to be saved in a file. We can achieve that with the `\o` command, we need to specify the filename in which the output of the queries will be saved. **NOTE: Once we use the `\o` command, all the queries will start appending the results in the provided file. To stop the behavior, you can use `\o` (without any filename) to stop appending the results to a file.** ``` # Start recording the output of queries to file \o filename # Stop recording the output of queries to file \o ```  ### Execute query from a file We can even execute queries stored in a file, we can use the `\i` command followed by the file name. This will load in the queries from the file and will display the output of each query. ``` \i file_name ```  ## Summary In this little article, we were able to explore the basics of the PGCLI package in python, with that package we can interact with the Postgres database that can be on your local server or somewhere remotely. With the PSQL-like shell provided by the PGCLI tool, we were able to write SQL queries, execute commands, write/read queries from files, etc. This tool is quite handy for people dealing with postgres databases and who need to view data or schema in that database. I've personally used this tool in my current internship where I needed to create a few SQL queries to fetch particular data related to various tables and relationships. It saved a lot of time with auto-completion and integration with the text editor for writing longer queries, especially with JOINS and nested queries. ## Conclusion This was my take and experience of using the PGCLI tool and exploring the various features it provides that can be leveraged for quick interactions with a Postgres database. Hopefully, you have learned something from this post, if you have any queries(NOT SQL queries) or feedback, please let me know in the comments or on my social handles. Thank you for reading and Happy Coding :) https://www.meetgor.com/posts/django-htmx-crud Introduction Gone are the days of writing Ajax requests with javascript, just add a few parameters to the HTML content tags and you will be ready for sending re Sun, 31 Jul 2022 00:00:00 UTC ## Introduction Gone are the days of writing Ajax requests with javascript, just add a few parameters to the HTML content tags and you will be ready for sending requests to your backend. So, we are going back in time and correcting the way we think of APIs and client/server-side rendering. We are referring to the [Hypermedia model](https://en.wikipedia.org/wiki/Hypermedia) for levering the server-side processing of data. Let's get our feets wet with this ancient but revolutionary methodology of development with [HTMX](https://htmx.org/). Yes, HTMX can be used for the API/server-side calls directly in the HTML. We will be exploring the basis of HTMX by creating a basic CRUD application. ## What is HTMX? The first question that might come up is what and why HTMX? Htmx is a great library, it's a javascript library but wait. It is a javascript library designed to allow us to write less or no javascript at all. It acts as a way to send AJAX requests without you writing any javascript. It uses native browser features directly from HTML. So, we can use HTMX to create interactive templates in our Django application. We can dynamically call and fetch data from the server by using simple HTML attributes like `hx-get`, `hx-post`, etc. We'll cover those in this article. You can check the source code used in this article on this [GitHub repository](https://github.com/Mr-Destructive/htmx-blog-django). ## Setup Django Project We'll be creating a Django project from scratch and designing a basic blog kind of app. We will be creating a quite simple project with a couple of apps like `user` for authentication and `article` for the CRUD part of our blog application. To set up a django project, we can run the following commands to quickly get up and running with a base django project. ``` mkdir htmx_blog python3 -m venv .venv source .venv/bin/activate pip install django django-admin startproject htmx_blog . ``` I have a base user model that I use for a simple authentication system in some basic django projects, you can define your own user app or get the app from [here](https://github.com/Mr-Destructive/django-todo/tree/master/user). So, that being said, we will be using the user model for the article model which we will be defined next. By creating a basic signup functionality, you are good to go! ### Create the Article app We will need at least an app to work with htmx as we will define models, views, and URLs later as we configure the htmx. ``` django-admin startapp article ``` After the app has been created, you can add those app labels into the `INSTALLED_APPS` config in the `settings.py` file. The `user` app and the `article` app need to be added to the installed apps for the django to pick those up for various contexts related to the project. ``` # htmx_blog/settings.py INSTALLED_APPS = [ ... ... ... 'article', 'user', ] ``` We are sone with the base setup, we also would require a few more configs for the proper working of the project. ### Setup Templates and Static files Templates will play an important role in the htmx part, so it is equally important to configure them properly before dabbling into the htmx and client-side rendering of data. I like to keep all the templates in a single folder in the `BASE_DIR` with separate sub-folders for specific apps. Also a single `static` folder with `css`, `js`, and `images` as the sub-folfers for a larger project. ``` mkdir templates static ``` Further, configure the created static and templates in the settings. ```python TEMPLATES = [ { 'BACKEND': 'django.template.backends.django.DjangoTemplates', 'DIRS': [os.path.join(BASE_DIR, "templates")], 'APP_DIRS': True, 'OPTIONS': { 'context_processors': [ 'django.template.context_processors.debug', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.contrib.messages.context_processors.messages', ], }, }, ] STATIC_URL = 'static/' STATICFILES_DIRS = [str(BASE_DIR/ "static")] STATIC_ROOT = BASE_DIR / "staticfiles" ``` ### Initial migration Run migration command for the user model and default model in the django project. ``` python manage.py makemigrations python manage.py migrate ``` So, this project will also include authentication simple registration, and login/logout routes. We will be using the default Django User model by creating an abstract user just in case we require any additional attributes. ## Setup HTMX We don't have to configure much for using HTMX as it is a javascript library, we can call it via a CDN or manually install it and link up the static javascript files. Either way, both are equally good, you may like the one I might like the other. If you already have a base template, you can simply put the below script inside the head tag of the template. This will make us the htmx attributes available. ```html <script src="https://unpkg.com/htmx.org@1.8.0"></script> ``` If you don't have a base template, you can create one by creating an HTML file inside the `templates` directory. The name can be anything but be careful for following up as it might be different for me. I will choose `base.html` as the template for this project. It will look something like as follows: ```html <!-- tempaltes/base.html --> <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>HTMX Blog</title> {% load static %} <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css" integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm" crossorigin="anonymous"> <script src="https://unpkg.com/htmx.org@1.8.0"></script> </head> <body > <nav> <h2>HTMX Blog</h2> <div class="navbar"> {% if user.is_authenticated %} <a class="nav-item nav-link" href="#"logout' %}"><button class="btn btn-link">Logout</button></a> {% else %} <a class="nav-item nav-link" href="#"login' %}"><button class="btn btn-link">Login</button></a> <a class="nav-item nav-link" href="#"register' %}"><button class="btn btn-link">Register</button></a> {% endif %} </div> </nav> {% block body %} {% endblock %} </body> </html> ``` I have a nav bar with my user authentication views, simply a login or signup button if the user is not logged in and a log-out button if the user is authenticated. We have added the htmx script file from the CDN just before the end of the head tag. We also have included the bootstrap CSS file for a decent UI which we will be creating in this post. That is one of the ways, htmx can be injected into an HTML template, you can even download the javascript file from the [htmx cdn](https://unpkg.com/browse/htmx.org/dist/). Further, this can be downloaded or pasted into your local folder and served as a static file or embedded directly into an HTML template. ## Defining Models We will start the tutorial by defining the model of the application we are creating. Here, we will create a simple Article model with a few parameters like `title`, `content`, `author`, etc. ```python from django.db import models from user.models import Profile class Article(models.Model): Article_Status = ( ("DRAFT", "Draft"), ("PUBLISHED", "Published"), ) title = models.CharField(max_length=128, unique=True) content = models.TextField() author = models.ForeignKey(Profile, on_delete=models.CASCADE) status = models.CharField( max_length=16, choices=Article_Status, default=Article_Status[0], ) def __str__(self): return self.title ``` In the above model `Article`, we have a few fields like `title` simple Character Field, `content` as a text field as it will be a large text as the post body, `author` which is a ForeignKey to the User Model. We also have the status, which is defined as a character field but with a few choices like `draft` or `published`, we can further modify this status as public or private. But just keeping it simple and easy to understand. The object reference name for this model is the title as we have defined in the dunder string method. So, that is a simple model created, we can now migrate the changes into the database for adding the tables and attributes. ``` python manage.py makemigrations python manage.py migrate ``` This will make migrations to the database i.e. convert the python model class into database tables and attributes. So, once the migration process is completed successfully, we can move into the crust of this article which is to actually design the views. In the next section, we will be utilizing the models in our views for representing the data on the templates. ## Creating Article Form Before diving into the views section, we need a few things like the Article Form, which will be a Django Model-based form. It will help us a lot in creating or updating the fields for the article model. We can define a form in a python file called `forms.py`, it's not necessary to keep your forms in the `forms.py` but if you have a lot of forms and models, it becomes a good practice to organize the components of our app. So, I'll be creating a new file inside of the `article` app called `forms.py` and defining the `ArticleForm`. ```python # article/forms.py from django import forms from .models import Article class ArticleForm(forms.ModelForm): class Meta: model = Article exclude = ( "created", "updated", "author", ) widgets = { "title": forms.TextInput( attrs={ "class": "form-control", "style": "max-width: 450px; align: center;", "placeholder": "Title", } ), "content": forms.Textarea( attrs={ "class": "form-control", "style": "max-width: 900px;", "placeholder": "Content", } ), } ``` So, the forms are inherited from the [ModelForm] which allows us to create forms based on our model. So, we specify the model name which in our case is `Article` and further we can have `exclude` or `fields` tuples. To exclude certain fields in the actual form, just parse the tuple of those attributes and if you want to only select a few attributes, you can specify the `fields` tuple and mention the required fields for the form. So, if we have a lot of things to be included in the form, we can specify only the attributes to be excluded with the `exclude` tuple. And if we have a lot of fields to be excluded, we can use the `fields` tuple to specify which attributes to use in the form. Let's take an example: For the above ArticleForm, if we wanted to specify the required fields to be included in the form, then we might use the `fields` tuple like below the rest will be not rendered in the form fields. ``` class ArticleForm(forms.ModelForm): class Meta: model = Article fields = ( "title", "content", "status", ) ``` Both of them can be used, it just depends on how many fields you have to exclude or include in the rendered form. We have also specified the `widgets` attribute which gives a bit more control on how we need to display the form in a template. So I have specified the type of input it needs to render like a simple text input for the title, text area for content, etc. The cool thing about this is it can automatically set these by knowing the type of field in the model, but sometimes it can be a bit undesired mostly with complex relationships and attributes. ## Creating Views Let's start creating views for creating, reading, updating, and deleting articles from the database. I will be using function-based views just because we are understanding the flow of how HTMX and Django can be integrated so we need to dive in deeper and understand the actual flow of the process. ### Create View So, creating articles seems like a good way to start off. We can create a simple function-based view which will initially load in an empty `ArticleForm` and if the request is `GET` we will render the form in the `create.html` template. If the request is `POST` which will be after we submit the form, we will validate the form and attach the current user as the author of the article and save the for instance which will create an article record and this object will be rendered to the detail template. ```python from django.shortcuts import render from .models import Article from .forms import ArticleForm def createArticle(request): form = ArticleForm() context = { 'form': form, } return render(request, 'articles/create.html', context) ``` #### Rendering the Form We are creating an empty instance of `ArticleForm` and rendering it in the template. So, this will render the empty form in the `create.html` template. ```html <!-- templates/articles/create.html --> {% extends 'base.html' %} {% block body %} <div hx-target="this" hx-swap="outerHTML"> <form> {% csrf_token %} {{ form.as_p }} <button hx-post="." class="btn btn-success" type="submit">Save</button> </form> </div> {% endblock %} ``` Now, here we are inheriting from the base template and creating a form tag in HTML with the `{{ form}}` for rendering the form fields, we finally have the `button` element for submitting the form. We have used the `hx-post` attribute. More on this in just a minute. So, this is we create a template for rendering the article form. We have used the `hx-post` attribute here, which will send a `POST` request to the current `URL` represented by `hx-post="."`. You might have noticed the `div` attributes, the `hx-target` and `hx-swap`, so these are some of the many attributes provided by the htmx library for controlling the reactivity of the requests made. The `hx-target` allow us to specify the element or tag to which the data will be rendered. The `hx-swap` goes hand-in-hand for specifying the target DOM like `innerHTML`, `outerHTML`, etc. You can see the various options on the [htmx docs](https://htmx.org/docs/#swapping). By specifying the `hx-swap` as `outerHTML`, we are saying to replace the entire element with the incoming content from the request which we will send with nearby request triggers. We need to map the view to a URL in order to get a good idea about the request and parsed content. We'll create a `create/` route and bind it to the `createArticle` view with the name `article-create`. ```python # article/urls.py from django.urls import path from . import views urlpatterns = [ path('create/', views.createArticle, name='article-create'), ] ``` This URL will be mapped to the global URL in the project, here we can simply specify the prefix for the URLs in the `article` app and include those URLs. ```python # htmx_blog/urls.py from django.contrib import admin from django.urls import path, include urlpatterns = [ path('admin/', admin.site.urls), path('user/', include('user.urls'), name='auth'), path('', include('article.urls'), name='home'), ] ``` Feel, free to add any other URL pattern like for instance, the article app is at `/` i.e. `127.0.01.:8000/`, you can add any other name like `127.0.0.1:8000/article/` by adding `path('article/', include('article.urls'))`.  So, finally, we are sending a `GET` request to the `127.0.0.1:8000/create/` and this will output the form. As we have a `POST` request embedded in the button inside the form, we will send the `POST` request to the same URL -> `127.0.0.1:8000/create/`. #### Submitting the Form Let's handle the `POST` request in the create view. ```python from django.shortcuts import render from .models import Article from .forms import ArticleForm def createArticle(request): form = ArticleForm(request.POST or None) if request.method == 'POST': if form.is_valid(): form.instance.author = request.user article = form.save() return render(request, 'articles/detail.html', {'article': article}) context = { 'form': form, } return render(request, 'articles/create.html', context) ``` **Simple explanation** - Create a form instance of ArticleForm with the request data or empty -> `ArticleForm(request.POST or None)` - If it's a POST request, validate and create the article, render the article object in `detail.html` template. - If it's a GET request, render the empty form in `create.html` There are a few changes in the view, instead of initializing the form to empty i.e. `ArticleForm()`, we are initializing with `ArticleForm(request.POST or None)`. This basically means that if we are having something in the `request.POST` dict, we will initialize the Form with that data or else an empty form instance. Next, we check if the request if `POST`, if it is then we check if the form is valid i.e. the form fields are not empty or if any other constraint on the model attributes is satisfied or not. If the form data is valid, we attach the author as the currently logged-in User/user who sent the request. Finally, we save the form which in turn creates the article record in the database. We then render the created article in the `detail.html` template which is not yet created. So, the `htmx-post` attribute has worked and it will send a post request to the same URL i.e. `127.0.0.1:8000/create` and this will again trigger the view `createArticle` this time we will have `request.POST` data. So, we will validate and save the form. ### Detail View The detail view is used for viewing the details of an article. This will be rendered after the article has been created or updated. This is quite simple, we need an `id` or `primary key(pk)` of an article and render the `title` and `content` of the article in the template. We pass in a primary key along with the request as a parameter to the view, the `pk` will be passed via the URL. We fetch the Article object with the id as the parsed `pk` and finally render the `detail.html` template with the article object. The `context['article']` can be accessed from the template to render the specific attributes like `title`, `content`, etc. ```python # article/views.py def detailArticle(request, pk): article = Article.objects.get(id=pk) context = {'article': article} return render(request, 'articles/detail.html', context) ``` We can now bind the view to a URL and parse the required parameter `pk` to the view. ```python from django.urls import path from . import views urlpatterns = [ path('create/', views.createArticle, name='article-create'), path('<int:pk>', views.detailArticle, name='article-detail'), ] ``` We have parsed the `pk` as `int` to the URL parameter, so for an article with id=4, the URL will be, `127.0.0.1:8000/4/`. We need to create the template for rendering the context from the `detailArticle` view. So, we create the `detail.html` in the `templates/articles` folder. We inherit the base template and render the `article.title` and the `article.content` with a linebreaks template filter so as to display the content properly. ```html <!-- templates/articles/detail.html --> {% extends 'base.html' %} {% block body %} <div id="article-card"> <h2>{{ article.title }} <p>{{ article.content|linebreaks|safe }}</p> <div> {% endblock %} ```  So, we can now use `createArticle` view as well as `detailArticle` view, this both are configured properly, so (CR) or CRUD is completed. We can add `listArticle` for listing out all the author's(logged-in user) articles. ### List View Listview of the articles is much similar to the detail view as it will return a list of articles rather than a single article. So in the `listArticle` view, we will return all the articles with the author as the user who sent the request/logged-in user. We will parse this object list into the template as `base.html` or `articles/list.html`. ```python # article/views.py def listArticle(request): articles = Article.objects.filter(author=request.user.id) context = { 'articles': articles, } return render(request, 'base.html', context) ``` We will add the URL route for this as the `/` route that is on `127.0.0.1:8000/` this is the base URL for the article app and is the route for the `listArticle` view. So, we will display the list of articles on the homepage. ```python # article/urls.py from django.urls import path from . import views urlpatterns = [ path('<int:pk>', views.detailArticle, name='article-detail'), path('create/', views.createArticle, name='article-create'), path('', views.listArticle, name='article-list'), ] ``` Let's create the template for the list view which will iterate over the articles and display the relevant data like the title and link to the article. ```html <!-- templates/articles/list.html --> <ul id="article-list"> {% for article in articles %} <li> <div class="card" style="width: 18rem;"> <div class="card-body"> <h5 class="card-title">{{ article.title }}</h5> <p class="card-text">{{ article.content|truncatewords:5 }}</p> <a href="#"article-detail' article.id %}" class="card-link">Read more</a> </div> </div> </li> {% endfor %} </ul> ``` We have used the `truncatewords:5` template filter for only displaying the content of the articles till the first 5 words as it is just a list view, we don't want to display every detail of the article here.  We can use this template to render in the `base.html` file. ```html <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>HTMX Blog</title> {% load static %} <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css" integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm" crossorigin="anonymous"> <script src="https://unpkg.com/htmx.org@1.8.0"></script> </head> <body hx-target="this" hx-swap="outerHTML" hx-headers='{"X-CSRFToken": "{{ csrf_token }}"}'> <nav> <h2>HTMX Blog</h2> <div class="navbar"> {% if user.is_authenticated %} <a class="nav-item nav-link" href="#"article-list' %}"><button class="btn btn-link">Home</button></a> <a class="nav-item nav-link" href="#"logout' %}"><button class="btn btn-link">Logout</button></a> {% else %} <a class="nav-item nav-link" href="#"login' %}"><button class="btn btn-link">Login</button></a> <a class="nav-item nav-link" href="#"register' %}"><button class="btn btn-link">Register</button></a> {% endif %} </div> </nav> {% block body %} <a href="#"article-create' %}"><button class="btn btn-success" >Create</button></a> {% include 'articles/list.html' %} {% endblock %} </body> </html> ``` We have now included the `list.html` template on the homepage and also added the `create` button as the link to the `article-create` URL. ### Delete View For deleting an article, we will simply rely on htmx for sending the request and on that request, we will delete the current article and render the updated list of articles. With the `deleteArticle` view, we will take in two parameters the request which is by default for a Django function-based view, and the primary key as `pk`. Again we will parse the `pk` from the URL. We will delete the article object and get the latest list of articles. Finally, render the updated list of articles in the base template which is our list view. ```python # article/views.py def deleteArticle(request, pk): Article.objects.get(id=pk).delete() articles = Article.objects.filter(author=request.user) context = {'article': articles} return render(request, "base.html", context) ``` We will add the `deleteArticle` into the URL patterns and call it `article-delete` with the URL of `delete/<int:pk>`. This will allow us to send a request to the URL `127.0.0.1:8000/delete/4` for deleting the article with id `4`. ```python # article/urls.py from django.urls import path from . import views urlpatterns = [ path('', views.listArticle, name='article-list'), path('<int:pk>', views.detailArticle, name='article-detail'), path('create/', views.createArticle, name='article-create'), path('delete/<int:pk>', views.deleteArticle, name='article-delete'), ] ``` In the delete view, the template is important as we want to send a request appropriately to the defined URL. To do that, we will have a form but it won't have any inputs as such just a button that indicates to delete the current article. We will add the `hx-delete` attribute as the URL to the `deleteArticle` view. with the id of the article. This will send a request to the `article-delete` URL which will, in turn, trigger the view with the given id and delete the article. We have added the `hx-confirm` attribute for showing a pop-up of confirmation of deleting the article. As you can see we have added a little script for adding `csrf_token` into the HTML, this is important in order to submit a form with a valid `CSRFToken`. ```html <!-- templates/article/delete.html --> <script> document.body.addEventListener('htmx:configRequest', (event) => { event.detail.headers['X-CSRFToken'] = '{{ csrf_token }}'; }) </script> <div > <form method="post" > {% csrf_token %} <button class="btn btn-danger" hx-delete="{% url 'article-delete' article.id %}" hx-confirm="Are you sure, You want to delete this article?" type="submit"> Delete </button> </form> </div> ``` Do you have a question like how do we access the `article.id`? we are not rendering the `delete.html` template from the view, so there is no context to pass. We will include this snippet into the detail view template, so as to have the option of deleting the current article. We will modify the `articles/detail.html` template and include the `delete.html` template. This includes simply adding an HTML template in the specified location. So, we will basically inject the delete form into the detail template. ```html {% extends 'base.html' %} {% block body %} <div hx-target="this" hx-swap="outerHTML"> <h2>{{ article.title }} {% include 'articles/delete.html' %} <p>{{ article.content|linebreaks|safe }}</p> <div> {% endblock %} ``` Hence, we will have a nice option to delete the article in the detail section, this can be placed anywhere but remember, we need to add the `hx-target="this"` and `hx-swap="outerHTML"` in the div so as to correctly swap the HTML content after the request has been made. ### Update View We can now move into the final piece of the CRUD i.e. `Update`. This will be similar to the `createArticle` with a couple of changes. We will parse parameters like `pk` to this view as well because we want to update a specific article. So, we will have to get the primary key of the article from the URL slug. Inside the `updateArticle` view, we will first grab the article object from the parsed primary key. We will have two kinds of requests here, one will be for fetching the `form` with the current article data, and the next request will be the `PUT` request for actually saving the changes in the article. The first request is simple as we need to parse the form data with the instance of the article object. We will call the `ArticleForm` with the instance of `article` this will load the data of the article into the form ready to render into the template. So once the `GET` request has been sent, we will render the template with the form pre-filled with the values of the article attributes. ```python # article/views.py def updateArticle(request, pk): article = Article.objects.get(id=pk) form = ArticleForm(instance=article) context = { 'form': form, 'article': article, } return render(request, 'articles/update.html', context) ``` We will create a template in the `templates/articles/` folder as `update.html` which will have a simple form for rendering the form fields and a button for sending a `PUT` request. We will render the `form` and then add a button element with the attribute `hx-put` for sending the `PUT` request to save changes to the article record. We will parse in the `article.id` for the primary key parameter to the view. ```html <!-- templates/articles/update.html --> <div hx-target="this" hx-swap="outerHTML"> <form> {% csrf_token %} {{ form.as_p }} <button hx-put="{% url 'article-update' article.id %}" type="submit">Update</button> </form> </div> ``` We are yet to link the `updateArticle` into the URLs. We will add the view `updateArticle` into the URLs with the name as `article-update` and `update/<int:pk` as the slug pattern. This URL pattern will trigger the `updateArticle` when we send an HTTP request to the `127.0.0.1:8000/update/4` for updating the article with id as `4`. ```python # article/urls.py from django.urls import path from . import views urlpatterns = [ path('', views.listArticle, name='article-list'), path('<int:pk>', views.detailArticle, name='article-detail'), path('create/', views.createArticle, name='article-create'), path('delete/<int:pk>', views.deleteArticle, name='article-delete'), path('update/<int:pk>', views.updateArticle, name='article-update'), ] ``` This is not done yet, we will need to handle the `PUT` request as well i.e. when the form details have been modified and we are about to save changes to the form data. So, we will check for the request method's type. If it is a `PUT` request, we will have to process a few things. ```python # article/views.py from django.http import QueryDict def updateArticle(request, pk): article = Article.objects.get(id=pk) if request.method == 'PUT': qd = QueryDict(request.body) form = ArticleForm(instance=article, data=qd) if form.is_valid(): article = form.save() return render(request, 'articles/detail.html', {'article': article}) form = ArticleForm(instance=article) context = { 'form': form, 'article': article, } return render(request, 'articles/update.html', context) ``` In the above `updateArticle` view, we have to check for a `PUT` request, if we are sending a `PUT` request, the form instance needs to be loaded from the request object. We use the `request.body` to access the data sent in the request. The incoming data received from the `request.body` object is not a valid format to parse it to the form instance, so we will parse it using `QueryDict`. This will allow us to modify the `request.body` object into valid python serializable data. So, we import the `QueryDict` from `django.http` module. We parse the data as the parameter to `QueryDict` and store it in a variable. We then have to get the `ArticleForm` for fetching the data as per the form details, so we parse the instance and also the `data` parameter. The instance is the article object and the data is the received form data which we have stored in `qd` as `QueryDict(request.body)`. This will load the new form data and then we can validate it the form. After we have verified the form details, we can save the form and this will update the article record. Thereby we can render the updated article in the `detail` view with the updated `article` object as the context.  So, this will set up the update view as well, we can now create, read, update, and delete an article instance with HTMX in templates and Django function-based views without writing any javascript. ## Summary We were able to create a basic CRUD application in Django with HTMX. We used simple function-based views to demonstrate the inner details of how we can work with HTMX and handle requests from the templates. By creating simple standalone templates, we can connect those together to make a fully functional and responsive webpage. The UI is not great but the purpose of this tutorial was to make a barebone CRUD app to work with the backend using HTMX, so hopefully, you would have got a good overview of how HTMX can be integrated into a Django application. Overall HTMX is a great library that can be used to enhance or even create a new web application for making the site responsive and without writing any javascript.  You can check out the source code for this project and blog on the [htmx-blog GitHub](https://github.com/Mr-Destructive/htmx-blog-django) repository. ## Conclusion From this post, we were able to understand the basics of HTMX and how we can integrate it into a Django application. Hopefully, you enjoyed the post, if you have any queries or feedback, please let me know in the comments or on my social handles. Thank you for reading. Happy Coding :) https://www.meetgor.com/posts/golang-anonymous-functions Introduction We have looked at the defer keyword in golang in the part of the series, in this section, we will understand how we can use anonymous functions in Sun, 24 Jul 2022 00:00:00 UTC ## Introduction We have looked at the defer keyword in golang in the [previous](https://www.meetgor.com/golang-defer/) part of the series, in this section, we will understand how we can use anonymous functions in golang. We will explore how to declare and use anonymous functions with a few examples. ## What are Anonymous Functions Anonymous functions are quite simple to understand, we don't define a function, we declare it and call it instantly. An anonymous function doesn't have a name so hence it is called an anonymous function. As a normal function it can take in parameters and return values. With anonymous functions, we can bind the operations to a variable or a constant as a literal(value). If an anonymous function takes in a parameter, it needs to be parsed immediately after the end of the function body. We will see how we define the syntax and specifications of the anonymous functions in golang. ## Simple Anonymous functions To create a simple anonymous function we use the same function syntax without giving it a name like `func() {}`, inside the function body i.e. `{}`, you can define the operations that need to be performed. Here, we have created an anonymous function that simply calls the `fmt.Println` function with a string. So, we have made things a little too much as we can even directly call the `fmt.Println` function from the main function, instead we have called an anonymous function that in turn calls the `fmt.Println` function. It might not make sense to use an anonymous function here, but it can be helpful for other complex tasks that you want to isolate the logic without creating a dedicated function for the process. ```go package main import "fmt" func main() { func() { fmt.Println("Hello, Anonymous functions") }() } ``` ``` go run anonymous_function.go Hello, Anonymous functions ``` So, this is how we create a basic anonymous function in golang, this can be further used for returning values from the function or passing parameters into the function and also assigning the function call to a variable or a constant. ## Assigning anonymous function to variables We can assign the call to the anonymous function to a variable or a constant and call the function as many times as we require. So, we can basically store the function logic in a variable and call it whenever we require the function with the `()` parenthesis as an indication to call the function. In the following example, we have used the variable `draw` to store the function call which prints `Drawing` with the help of the `fmt.Println` function. The draw variable now contains the function and not its value. So be careful here, the anonymous function which we have defined as the `draw` variable's literal value, it's like we are giving a name to this anonymous function and the name will be the variable name so we have created the function `draw` which is an anonymous function stored in a variable. ```go package main import "fmt" func main() { draw := func() { fmt.Println("Drawing") } draw() draw() } ``` ``` go run anonymous_function.go Drawing Drawing ``` We see that we call the variable `draw` as a function call by appending `()` parenthesis to it as `draw()` this thereby calls the anonymous function stored inside the variable value. The main thing to note here is that we are not adding `()` at the time of declaring the anonymous function, as it will call the function directly. The problem then will arise because the function is not returning anything so we can't assign it to a variable. ```go draw := func() { fmt.Println("Drawing") }() ``` ``` functions/anonymous_functions.go:10:11: func() {…}() (no value) used as value ``` So, we are trying to assign a variable to a function call that has no return value. This has no value, not even nil, so we get an error for assigning a variable to nothing. Though if you had a return value from the function, we can directly assign the function call to the variable as it has returned a valid value defined in the function literal. ## Parsing parameters We can even parse parameters to the anonymous functions as any other normal function. We define the name of the variable followed by the type of the variable in the parenthesis to use these parameters inside the anonymous function. We need to keep in mind that these function parameters can either be passed with the variable call or directly at the time of defining the function. In the below example, we have created a variable `call` and are assigning it to an anonymous function that takes in a parameter `name` which is a `string`, and prints some text to the console. So, we call the funciton `call` with a parameter as a string, as we have `"Meet"` and `person := "Chris"` as a string passed to the `call` function. ```go package main import "fmt" func main() { call := func(name string) { fmt.Println("Calling,", name) } call("Meet") person := "Chris" call(person) } ``` ``` go run anonymous_function.go Calling, Meet Calling, Chris ``` Here, we can see that the function `call` prints the text that we have defined to call the `fmt.Println` function. We parse the function with the string literal as the anonymous function takes a parameter in the form of a string. We can parse multiple parameters to the anonymous function as we can with the normal function like slices, maps, arrays, struct, etc. ## Returning values We can even return values from the anonymous function if we want to instantly call the function and save the `returned` value in the variable. We can return single or multiple values as per our requirements just like any normal function in golang. ```go package main import "fmt" func main() { is_divisible := func(x int, y int) bool{ var res bool if x % y == 0 { res = true } else{ res = false } fmt.Println(res) return res } fmt.Printf("%T ", is_divisible) is_divisible(10, 5) is_divisible(33, 5) divisblity_check := is_divisible(45, 5) fmt.Printf("%T : %t ", divisblity_check, divisblity_check) } ``` ``` go run anonymous_function.go func(int, int) bool true false bool : true ``` As we can see the function has returned a boolean value and we store it in a variable `divisblity_check`, the variable which contains the function i.e. `is_divisible` can then be called, and thereby we get the returned value in the variable as a boolean as we print the type and the value of the variable `divisblity_check`. We also can see that the type of the variable `is_divisible` is of type function literal which takes `int, int` and has a return value as `bool`. We can also do one more interesting here, which we were restricted previously in the above examples. We can directly call the function and store it as a value rather than the function itself. So, we can only use the value returned from the function but can't call the function later. ```go is_divisible := func(x int, y int) bool{ var res bool if x % y == 0 { res = true } else{ res = false } fmt.Println(res) return res }(13, 4) fmt.Printf("%T ", is_divisible) fmt.Printf(is_divisible) ``` ``` go run anonymous_function.go bool false ``` So, in the above-modified example, we have passed in the parameter instead of a callable function. This will store the returned value of the function call. So, we will store the boolean value in the `is_divisible` and we will have to pass the integer values to the function which we have parsed as `(13, 4)` to the anonymous function call. In the below example, we have created an anonymous function that takes in three parameters like `(string, int, string)` and returns a string. We have used `fmt.Sprintf` function to format the variable and store it in a variable, we then return the string. This anonymous function is then directly called and we store the returned value instead of the function. So, in this example, the `format_email` variable will store a string instead of acting it as a function as a callable object. ```go package main import "fmt" func main() { format_email := func(name string, age int, company string) string{ email := fmt.Sprintf("%s.%d@%s.com", name, age, company) return email }("john", 25, "gophersoft") fmt.Println(format_email) fmt.Printf("%T ",format_email) } ``` ``` go run anonymous_function.go john.25@gophersoft.com string ``` As we can see the `format_email` variable only returns a string instead of a callable object. We have directly parsed the parameters to the anonymous function and hence it instantly calls it and we return the string. ## Passing Anonymous function to function parameters We can even pass an anonymous function to a parameter to a function. This can be quite helpful for writing some simple logic inside a complex script. We can pass the function type as a parameter to the parameter and theere we can parse the actual data and perform the desired operation. The below example is a bit of code to write but makes a lot of sense to understand anonymous functions in golang. The function `get_csv` is a function which takes in three parameters `int, string, func(string)[] string`. The third parameter is a function literal that takes in a string and returns a slice of string. So, we are basically converting a string `"kevin,21,john,33"` into a slice of the slice like `[[kevin 21] [john 33]]`, this is basically separating values with `,` comma as the delimiter and then processing slices to create a single slice. ```go package main import "fmt" import "strings" func get_csv(index int, str string, t func(csv string)[]string) [][]string{ s := t(str) var res [][]string for i := 1; i<len(s); i+=2 { var data []string data = append(data, s[i-1], s[i]) res = append(res, data) } return res } func main() { csv_slice := func (csv string) []string{ return strings.Split(csv, ",") } csv_data := get_csv(2,"kevin,21,john,33,george,24", csv_slice) fmt.Println(csv_data) for _, s := range csv_data{ fmt.Printf("%s - %s ",s[0],s[1]) } ``` ``` go run functions/anonymous_function.go [[kevin 21] [john 33] [george 24]] kevin - 21 john - 33 george - 24 ``` Let's break down the code one by one, we will start with the main function, where we have `csv_slice` as a function literal and is an anonymous function that takes in a string and returns a slice of string. The function returns a call to the function [strings.Split](https://pkg.go.dev/strings#Split) taking in the string from the function parameter. We then call the function `get_csv` with parameters `(2, "kevin,21....", csv_slice)`, this function is defined outside the main. The function takes in three parameters as discussed and parsed from the main function and it returns a slice of type string. So, inside the `get_csv` function, we define `s` as the function call to `t(str)` which if you look carefully is a function call to `csv_slice` with parameter as a string. This function call returns a slice of strings separated by `,`. So that's all the logic we need to understand anonymous function from parameters. We have used a function literal to call the function from another function. In this case, the funciton is an anonymous function assigned to a variable. ``` "kevin,21,john,33,george,24" || \/ [kevin 21 john 33 george 24] || \/ [[kevin 21] [john 33] [george 24]] ``` Further, after we have `s` which would look like `[kevin 21 john 33 george 24]` as each individual element. Thereafter we create an empty [slice of slice](https://www.geeksforgeeks.org/slice-of-slices-in-golang/) string as `res` and operate a loop through the slice jumping 2 indexes at a time. Why? because we want a slice of two elements combined. So, we also create a slice of string named `data` and we add the two components to it like `[kevin 21]` with the [append](https://pkg.go.dev/builtin#append) function, and this slice is appended to a slice of slices `res` as `[[kevin 21]]` thereby iterating over the entire slice and creating the desired slice data. We return the `res` from the function. This get's us back to the main function which simply iterates over the slice and we get the desired data from it. So, this is how we convert an extremely easy task to extremely complicated code :) That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/mr-destructive/100-days-of-golang/tree/main/scripts/functions/anonymous_function.go) GitHub repository. ## Conclusion That is it from this part, we covered a little bit about anonymous functions in golang. Anonymous functions are simply function literals that can be used to do a lot of quick operations without needing to write an explicit function in the program. Further, in the next part look into closures which are a bit complex to understand and require the understanding of anonymous functions. Thank you for reading, if you have any queries, feedback, or questions, you can freely ask me on my social handles. Happy Coding :) https://www.meetgor.com/posts/django-deploy-railway-nixpacks Introduction We have seen how to deploy a Django application on railway app in the of the . We deployed the django project using the Heroku Buildpacks under the Sat, 16 Jul 2022 00:00:00 UTC ## Introduction We have seen how to deploy a Django application on railway app in the [previous article](https://www.meetgor.com/django-deploy-railway/) of the [series](https://www.meetgor.com/series/django-deployment). We deployed the django project using the Heroku Buildpacks under the hood. The railway app provides a couple of options on how to build your web application. Currently, there are three options, one of which is the [heroku buildpacks](https://devcenter.heroku.com/articles/heroku-20-stack) which is the default one, second, we have the nixpacks which we will see today, and the third is the [Paketo buildpack](https://paketo.io/). ## What is a Buildpack? A buildpack is a set of programs that turns your source code into a container image. So it is basically a tool for converting your application into a deployment-ready state with the help of containerization technology. Buildpacks allow us to extract away the steps for deploying an application. The layer of abstraction for converting a source code into a deployable code/container is played by the build packs. ### Benefits of buildpacks Buildpacks as we discussed act as a layer of abstraction from converting source code into the deployable containers, so it helps in avoiding manually writing dependencies and installing them. But on top of that, they can even detect low-level changes to the source code, i.e. if a dependency is changed, it would know which dependencies are altered and which components can be fetched from the cache. Caching is a great property of buildpacks which enhances the performance and deployment time. Since we can specify the commands for the build process, the buildpacks are customizable and hence provide a solid foundation for professional applications. For more references on Buildpacks, you can follow up with the great article on [dockerfile vs buildpacks](https://technology.doximity.com/articles/buildpacks-vs-dockerfiles). ## What are Nixpacks? [Nixpacks](https://nixpacks.com/docs) are quite similar to buildpacks, but they have their own set of technologies used in managing and installing builds for the application. Nixpacks as the name suggests uses [Nix Packages](https://search.nixos.org/packages) for creating and installing dependencies and [Docker](https://www.docker.com/) for building images and running containers. Nixpacks are quite cool as most of the applications require little or almost no configuration. You don't need a lot of knowledge of docker or nix technologies, everything is abstracted for you. Nixpacks uses nix packages for installing the runtime environment and the dependencies for the applications. It also caches support for detecting the core modules or packages in the runtime, so it gets a huge boost in deployment performance. Minimal deployment time, simultaneously makes it developer friendly and improves the quality of the application. - Abstracted technologies like Nix and Docker. - Caching of dependencies and steps to build. - Customizable at most of the steps. - Extensible and Developer friendly. ## Installing Nixpacks Nixpacks are the [nix packges](https://search.nixos.org/packages) which are used with the source code to create a buildpack of their own. The nix packages take in the source code of your application and convert it into a simple OCI image e.g. Docker image that can be run on various environments. It is similar to buildpacks but it's not the same instead it is better and uses a different ecosystem. Railway has a cool [CLI](https://nixpacks.com/docs/cli) tool for creating nixpacks on your local system. You can install the nixpack cli from the official documentation site. There are a couple of ways to install them on your system. You can install the nixpack with the curl command in your terminal which is one of the simplest ways to install it. ``` curl -fsSL https://raw.githubusercontent.com/railwayapp/nixpacks/master/install.sh | bash ```  Once it has been installed we can check the status if nixpacks was successfully installed or not. ``` nixpacks --version OR nixpacks ``` So that's how we install nixpacks CLI into the system. Now, we can move into the configuration required for creating a nixpack from the Django application. ## Creating a Django Project So, I assume here, you have your django project ready. You can pick any django project and configure it as mentioned below. ### Creating requirements file It is a good practice to include the `requirements.txt` file in any python based applications. It becomes really easy to set up and give the project a spin. It's often the case, that we have to use a virtual environment for creating a pip file, otherwise, we might conflict and mix up the globally installed packages with project-specific dependencies. ``` pip freeze > requirements.txt ``` This will create a `requirements.txt` file in the current folder, which will contain the list of all the dependencies with the version mentioned in. We also have other package management system like [pipenv](https://www.meetgor.com/pipenv-intro/) or [poetry](https://python-poetry.org/). ## Creating a Nixpack for the project Finally, we can now start creating nixpacks from the source code of the django application. So, make sure the Django project is running on your local system. Set up a virtualenv and database as per your local environment. We will be using the build command for creating the nixpack of our django project. ``` nixpacks build . ``` <video width="800" height="450" controls> <source src="https://res.cloudinary.com/techstructive-blog/video/upload/v1657881139/blog-media/nixpacks-demo-base.mp4" type="video/mp4"> </video> This is the simplest of commands you can run to create a nixpack. You need to be in the source file where all your files are located. In Django, we call it the `BASE_DIR`, the same folder where your `manage.py` resides. So, be on that path and run the above command. It would pick up a few things from the source code itself. There are also some default values picked up after analyzing the source code, for example in the case if it detects the django project, it attaches the `python manage.py migrate && gunicorn <project_name>.wsgi` command as the default build command. Similarly, there are other default options like the version of the runtime, installation steps, etc. We can see that the port is not accessible from the docker container that we ran it is because of two reasons either we have not exposed the port or the port is not bound with gunicorn. We will see that configuration later, it's quite easy to bind the local port to the docker container. ### Steps for creating Nixpacks Now, we need to understand the process of the creation of nixpacks. There were several stages that you can see in the above video. 1. Planning 2. Building #### Planning Phase The first step was quite important, to understand the runtime environment. This is critical because it can decide how our application installs, runs, or even performs in the actual container environment. This is just the phase where we define the sets of commands or nix packages to be used for the building of the application. You would get a clear look in the nixpack cli, it gives a list of the planned instructions before the build process starts in the build command. You can get the plan before building the application by using the `plan` command in the CLI. ``` nixpacks plan . ```   So, this gives a `nix` output of all the configuration needed to build and install the application and it's dependencies. When you would have built the application i.e. the base build command, it also shows in the CLI, some useful information before moving to the build step.  So the list of the following procedure is listed in the planning phase of building the application. - Packages/base runtime environment - Installation commands - Build commands - Commands to Start Application In the case of django, the following correspond to the planning attributes. - Python version as the application runtime. - Installing dependencies via pip/pipenv/poetry - Building the Django app (collecting static files, setting database, etc) - Running the Django app (gunicorn/Nginx web server to run the app) So, hopefully, this gives you a better understanding of what is going on in the nixpack CLI. We are trying to automate the process of deployment and building the application for repeated deployments i.e. in a continuous integration/delivery system. #### Building Phase This is the phase, where actual installation, setup, and configuration takes place at the application level. In the build phase, we have several layers where things happen like installing the core packages for the application, installing dependencies, setting/generating up the necessary steps for the proper running of the application, and finally steps to run the application. This process creates an OSI(a standard used in containers/virtualization technology) image of the application. We can run the image and thus create a container of the application. You will need docker or any other containerization tool for the building of images and running containers.  For the Django application, we have several steps of the application to be followed. We can use python as the base runtime environment, install the python packages, perform other commands like setting up admin accounts, static/media files, database setup, and finally run the steps to run the application at the container level. - Installing python as a [Nix Package](https://search.nixos.org/packages?channel=22.05&show=python38&from=0&size=50&sort=relevance&type=packages&query=python) - Installing all python packages provided in the `requirements.txt`/`Pipfile`/`pyproject.toml` files. - Call commands from the environment to set up the applications like `createsuperuser`, `collectstatic`, `makemigrations`, pull data, management commands, etc. There are a lot of things that can be done here depending on the application. - Finally, the step to run the Django app, usually gunicorn/Nginx server is used for running the django application on a port with the host. So, this is what the build phase does, this is the whole and soul of the nixpacks. We literally do every installation setup and configuration of the application. Though the planning phase is equally important, a single missing data can ruin the build phase. ### Understanding the build phase for Django We ran the build command for creating the nixpack build, this started with planning the application configuration and then building up the application. This build phase was also divided into further processes like installing, running commands, copying actual source code to an image, and all the docker-related stuff that is required to create an image for a django application.  So, we have used the Debian nixpack, which sets as the base runtime for the application. Railway provides a [package](https://github.com/railwayapp/nixpacks/pkgs/container/nixpacks) of the Debian image as the base runtime for our application. This is where we will run all the build processes on. This Debian image will be used for installing all types of dependencies and layers of language-specific runtime installation in the form of [nix packages](https://search.nixos.org/packages). Now, we have the base image of debian, the nixpack build command actually starts executing the `Dockerfile`, this is an auto-generated step after the planning phase. So, with the instructions in `Dockerfile`, steps are executed one after other just as a normal Docker image build. This will generate the image for the application and after a while, because this process takes a while on the first iteration locally, after the build process has been completed, it will give a container name for you to run. This can be used to test the application locally, in production, the container is instantly created after the image has been generated. ``` RUN: docker run -it <container_id_or_name> ```  This is the command for running your application, this marks the end of the build process and also the build command provided with the CLI. ### Build Command Parameters The build command in the nixpacks CLI provides a few parameters or arguments to customize how to output the result and build the application, you can definitely provide the configuration in the application source code itself, but it is nicer to have it locally before deploying the application. #### Give a name to the Nixpack Image/Container The first parameter which might be helpful is to provide a name to the application at the build time. This becomes useful for running the container, this helps in avoiding long container names and giving a context of the nixpack. ``` nixpacks build . --name <project_name> OR nixpacks build . -n <project_name> ```   This gives a name to the image which has been built. Thereby providing a better context for the user to run the image and create a container out of it. #### Output the Built Image to a folder This is the command that can output the built application into a provided folder. This parameter will not run the docker step thereafter; hence, no image is created if you provide an output folder. Though the folder will contain the `Dockerfile` and `environment.nix` files for creating the image and running the container. **Make sure the output folder is NOT in the application folder itself, it will result in errors.** The output command will not create an image but the process will be definitely executed in order to generate the `Dockerfile` and `environment.nix` files. ``` nixpacks build . --out ../blog_image OR nixpacks build . -o ../blog_image ```  **Dockerfile**  **environment.nix File**  So, this will output the built application into the provided path. The output folder should necessarily be out of the application folder as it makes no sense to output in the same folder as the application since the nixpacks CLI will consider the folder as the application folder. ### Provide a Install/Build/Start Command We can provide the commands at the install phase/build/start phase of the application to the build command in order to build the app with non-default or custom commands. This will add up to the docker steps that will involve making the build for the application. ``` nixpacks build . --build-cmd 'python manage.py collectstatic' OR nixpacks build . -b 'python manage.py collectstatic' ```  These kinds of parameters can be passed similarly for `install-cmd` and `start-cmd` as `-i` and `-s` respectively. We can further chain up the commands and customize the build process as per the application's requirements. ### Providing environment variables to image The environment variable can be passed to the build command for parsing to the application. This can be used for parsing additional or optional environment variables to the application image. ``` nixpacks build . --env 'NAME=VALUE' nixpack build . --env 'DATABASE_URL=postgres://postgres:postgres@localhost:5432/techstructive_blog' ```   Here, we provide the environment variable `DATABASE_URL` to the build command and this is parsed to the application image. Thereby when the image is run as a container, it is parsed as a normal environment variable and thereby is available for utilization from the application. For further references on the build command arguments, you can follow the [documentation of nixpack](https://nixpacks.com/docs/cli) by railway app. ### Creating a Procfile This is important for telling any buildpack in this case nixpacks to understand the process to start for this web application. For django it is simply to add the web process like to mention the `wsgi` app with the project name. We can use the gunicorn as the web server in production. ``` # Procfile web: gunicorn <django_project_name>.wsgi ``` This is the Procfile, this is a file type without the extension. So, this is a typical Django application Procfile, you can also use the other variants of Procfile for applying migration for every web process start-up. ``` # Procfile web: python manage.py migrate && gunicorn <django_project_name>.wsgi ``` As we saw in the base build command, the local server was not able to listen to the gunicorn server in the container, so we need to bind the gunicorn server to the local port. ``` # Procfile web: python manage.py migrate && gunicorn <django_project_name>.wsgi -b :8000 ``` So, we use the `-b` option in the gunicorn command to bind the port in the container to the port in the local machine. Now, if we build the application and forward the port to the `8000` port in the local machine, we will see our application running <video width="800" height="450" controls> <source src="https://res.cloudinary.com/techstructive-blog/video/upload/v1657964597/blog-media/nixpacks-local-bind.mp4" type="video/mp4"> </video> ``` docker run -p 8000:8000 -it <container_id> ``` ### Specifying the Python version This is created for specifying the python version for building the Django application or any other python app. ``` # .python-version 3.10 ``` Save the `.python-version` file with just the python version like `3.6`, `3.10`, etc. and this will be picked by the build command while creating the build image. ## Deploying the Django Application After we looked at the nix picks specifications, we can now deploy our Django application with nixpacks on Railway. So, you can follow up with the [Railway Deployment]() Article for setting up your Django app for deployment at the railway. This usually involves a few steps like creating Procfile(not necessary but recommended), requirements.txt(necessary to pull dependencies), and the python version which is chosen as `3.8` as default. The further steps are to create a GitHub repository to link with the Railway app and create a PostgreSQL database service on the railway platform. ### Create configuration files As we have seen we will require a `requirements.txt` file, `Pipfile` or a `pyproject.toml` file for listing out and installing dependencies for our django application. This can be done with various commands like: ``` # For requirements.txt and virtualenv pip freeze > requirements.txt # Autogenerated Pipfile for pipenv # Autogenerated pyproject.toml for poetry ``` So, this file should be present on the base directory of the django application in order for the nixpack to pick up and install the python packages. Also, for customization of the start command in the build process, you can create a `Procfile` as discussed earlier in order to run commands to start the Django web server. The python version can be specified with the `.python-version` file with just the version name as `3.9`, `3.10`, etc. OR we can add an environment variable `NIXPACKS_PYTHON_VERSION` to the python version we want. ### Create and Linkup a GitHub repository for existing Django projects We can create a GitHub repository and link up the project to the Railway platform and thereby creating an automated build for every push. The below video will explain how to set up the GitHub repository for the Railway app. <video width="800" height="450" controls> <source src="https://res.cloudinary.com/techstructive-blog/video/upload/v1652970864/blog-media/django-deployment/railway_project_init.webm" type="video/mp4"> </video> <video width="800" height="450" controls> <source src="https://res.cloudinary.com/techstructive-blog/video/upload/v1657965284/blog-media/railway-platform-github.mp4" type="video/mp4"> </video> ### Setup environment variables We can use `python-environ` to set up environment variables in a Django application, we will require these environment variables for attributes like `SECRET_KEY`, `DATABASE_URL`, email-stuff, etc. These are quite handy to avoid leaking sensitive information to the open source project on GitHub. You can install the `python-environ` package with pip or any other package management tool as follows: ``` pip install python-environ ``` After installing the package, we can set up the environment variable in the settings file. ``` python # <project_name>/settings.py import os from dotenv import load_dotenv BASE_DIR = Path(__file__).resolve().parent.parent load_dotenv(os.path.join(BASE_DIR, ".env")) ``` After loading the environment variables, we can access them with `os.env("ENV_NAME", default="")`, this will load the environment variable with the name or we can provide a default value. ### Attach a PostgreSQL database service You can add a PostgreSQL database service to your Django Railway app by attaching a service. This will add a new service along with the django application, so these two act as different entities within a railway project. <video width="800" height="450" controls> <source src="https://res.cloudinary.com/techstructive-blog/video/upload/v1652963718/blog-media/django-deployment/postgres_spinup_railway_d2xkpt.mp4" type="video/mp4"> </video> You can then access the `DATABASE_URL` from the connect settings and copy the database URL and set it as an environment variable in the django railway project service. This will link up the Django app and the PostgreSQL database. While setting it up locally, you can use the `.env` file and add the environment variable there. ``` # environment variable DATABASE_URL=postgres://username:password@hostname:port/db_name # local database postgres DATABASE_URL=postgres://postgres:postgres@localhost:5432/db_name ``` <video width="800" height="450" controls> <source src="https://res.cloudinary.com/techstructive-blog/video/upload/v1657964943/blog-media/railway-postgres-spinup.mp4" type="video/mp4"> </video> This will set up the database environment variable and you can access these from the settings.py file with the `env.db` function as follows: ```python env.db("DATABASE_URL", default=<local_database_url>) ``` So, we can finally use the database from the Railway app in our Django application once the environment variable is correctly used. ### Choose the Buildpack We can choose a buildpack for our Django application in the Railway platform, we have options like 1. Heroku Buildpack 2. Railway Nixpacks 3. Paketo Buildpack As of the writing of the article, on 16th July 21, the Railway has made `Nixpacks` the default buildpack for an application :) It was the `Heroku` Buildpack as a default earlier. So, that is a cool thing, you can toggle these settings for choosing the buildpacks from the project settings. Railway Dashboard Choose BuildPack <video width="800" height="450" controls> <source src="https://res.cloudinary.com/techstructive-blog/video/upload/v1657964699/blog-media/nixpacks-railway-dashboard.mp4" type="video/mp4"> </video> ### Deploy to Railway with Nixpacks Now, we have seen how to set up the nixpacks, we had the Postgres database setup, and we can finally deploy our application to the railway platform with nixpacks. <video width="800" height="450" controls> <source src="https://res.cloudinary.com/techstructive-blog/video/upload/v1657965560/blog-media/railway-nixpacks-deploy.mp4" type="video/mp4"> </video> So, we simply can configure the source code or include the `environment.nix` file into the source code for desired behavior. The nixpack selection can be done based on the source code or the presence of `environment.nix` and that's why we can rely on expected behaviors from the deployment builds. ## Summary So, nixpacks is a great way to deploy an application, for me it's an automated docker deployment, it basically creates docker images of the application and runs it with the appropriate environment. There is a lot of language support on nixpacks currently on Railway, you can check them out on the official website. Every programming language has specific requirements for managing or installing dependencies and packages, the nixpacks manage them automatically for us. ## Conclusion So, from this post of the [Django Deployment Series](https://www.meetgor.com/series/django-deployment), we were able to understand how to deploy a Django application on the Railway app with Nixpacks which are a very intuitive way to deploy apps. We covered what are nixpacks, the process of building an application with nixpacks, and deploying a existing, new Django project on the railway with nixpacks. We also explored the various commands provided in the nixpacks CLI to build. plan a Django application. Hopefully, you were able to understand the concept of nixpacks and how they can automate the process of containerization and deployment. Thank you for reading, if you have any queries or feedback, you can leave them down in the comments or freely drop them on social media. Happy Coding :) https://www.meetgor.com/posts/neovim-vimscript-to-lua Introduction It has been a while since I have written a Vim article. Finally, I got some ideas after configuring my Neovim setup for Lua. I recently migrated to Sat, 09 Jul 2022 00:00:00 UTC ## Introduction It has been a while since I have written a Vim article. Finally, I got some ideas after configuring my Neovim setup for Lua. I recently migrated to Ubuntu a couple of months back and it has been a cool change from Windows 7! In this article, we'll see how you can set up neovim for Lua. Since Neovim 0.5, it supports lua out of the box, so in the recent release 0.7, it added more native support to lua making it a lot easier to configure and play with neovim. So, we will see how we can use lua to convert all the 200 liner vimscript into lua (We can even have packages and modules:) We will cover how to configure your keymaps, pull up all the plugins, vim options, and other customizations. ## Why move to Lua? I have used Vimscript for quite a while now, configured it as per my needs, and also made a few plugins like [frontmatter](https://github.com/Mr-Destructive/frontmatter.vim), [dj.vim](https://github.com/Mr-Destructive/dj.vim), and [commenter](https://github.com/Mr-Destructive/commenter.vim) which are quite clunky and not robust in terms of usage and customizability. Vimscript is good but it's a bit messy when you want extreme customization. I recently wanted to go full Neovim, I was kind of stuck in Vimscript and some of my plugins work for me but it might not work for others, it might be a piece of gibberish vimscript dumped. So, Why not have full native experience in Neovim if you can. It has now baked-in support for LSP, Debugging, Autocommands, and so much more. If you have Neovim 0.5+ you should be good to go full lua. ## Backup Neovim Config Firstly, keep your original neovim/vim init files safe, back them up, make a copy and save it with a different name like `nvim_config.vim`. If you already have all your config files backed up as an ansible script or dotfiles GitHub repository, you can proceed ahead. But don't keep the `init.vim` file as it is in the `~/.config/nvim` folder, it might override after we configure the lua scripts. ## Basic Configuration So, I assume you have set up Neovim, If not you need to follow some simple steps like downloading the package and making sure your neovim environment is working with vimscript first. The [Neovim Wiki](https://github.com/neovim/neovim/wiki/Installing-Neovim) provides great documentation on how to install neovim on various platforms using different methods. After your neovim is set up and you have a basic configuration, you can now start to migrate into lua. Create a `init.lua` file in the same path as your `init.lua` file is i.e. at `~/.config/nvim` or `~/AppData/Local/nvim/` for Windows. That's why it is recommended to keep the initial configuration vimscript file in a safe place. While migrating from vimscript to lua, once the lua file is created and the next time you restart neovim, the default settings will be from `init.lua` and not `init.vim`, so be prepared. Firstly, you need to configure some options like `number`, `syntax highlighting`, `tabs`, and some `keymaps` of course. We can use the `vim.opt` method to set options in vim using lua syntax. So, certain corresponding vim options would be converted as follows: If you have the following kind of settings in your vimrc or init.vim: ```vimscript -- vimscript set number set tabstop=4 set shiftwidth=4 set softtabstop=0 set expandtab set noswapfile ``` The above settings are migrated into lua syntax as follows: ```lua --lua vim.opt.number = true vim.opt.tabstop = 4 vim.opt.shiftwidth = 4 vim.opt.softtabstop = 0 vim.opt.expandtab = true vim.opt.swapfile = false ``` You can set other options in your config file accordingly. If you get sick of writing `vim.opt.` again and again, you can use a variable set to `vim.opt` and then access that variable to set the options. Something of the lines as below: ```lua local set = vim.opt set.number = true set.tabstop = 4 set.shiftwidth = 4 set.softtabstop = 0 set.expandtab = true set.swapfile = false ``` We can create a variable in lua like `local variable_name = something` so, we have created a variable `set` which is assigned to the value of `vim.opt` which is a method(function) in lua to set the options from the vimscript environment. Finally, access the `set` keyword to set the options. Using the `set` word is not necessary, you can use whatever you want. After setting up the basic options, you can source the file with `:so %` from the command mode. Just normally as you source the config files. ### Using Lua in Command Mode We can use the lua functions or any other commands from the command mode in neovim using the lua command. Just prefix the command with `:lua` and after that, you can use lua syntax like function calling or setting variables, logging things, etc.  ## Adding Keymaps Now, that we have some basic config setup, we can quickly get the keymaps. It's not that hard to make keymaps to set up in lua. To create keymaps, we have two options. One is compatible with Neovim and the other is compatible with Vim as well. 1. vim.keymap.set OR 2. vim.api.nvim_set_keymap Personally, I don't see a difference in terms of usage but [vim.keymap.set](https://github.com/neovim/neovim/blob/master/runtime/lua/vim/keymap.lua#L51) is a wrapper around [nvim_set_keymap](https://github.com/neovim/neovim/blob/master/src/nvim/api/vim.c#L1451). So, it is really a matter of personal preference which you want to use. So, both the functions have quite similar parameters: ```lua vim.keymap.set({mode}, {lhs}, {rhs}, {options}) vim.api.nvim_set_keymap({mode}, {lhs}, {rhs}, {options}) ``` The advantage of `vim.keymap.set` over `vim.api.nvim_set_keymap` is that it allows directly calling lua functions rather than calling vimscripty way like `:lua function()`, so we directly can use lua code in the RHS part of the function parameter. Let's take a basic example mapping: ``` vim.keymap.set('n', 'Y', 'yy', {noremap = false}) ``` Here, we have mapped `Shift+y` with the keys `yy` in `n`ormal mode. The first argument is the mode, it can be a single-mode like `'n'`, `'v'`, `'i'`, etc., or a multi-mode table like `{'n', 'v'}`, `{'v', 'i'}`, etc. The next parameter is also a string but it should be the key for triggering the mapping. In this case, we have used `Y` which is `Shift + y`, it can be any key shortcut you want to map. The third parameter is the string which will be the command to be executed after the key has been used. Here we have used the keys `yy`, if the map is from a command mode, you will be using something like `':commands_to_be executed'` as the third parameter. The fourth parameter which is optional can contain [special arguments](https://neovim.io/doc/user/api.html#:~:text=nvim_set_keymap(%7Bmode%7D%2C%20%7Blhs%7D%2C%20%7Brhs%7D%2C%20%7B*opts%7D)%20%20%20%20%20%20%20%20%20%20%20%20%20*nvim_set_keymap()*). We have set a default option which is `noremap` as true, the options are not string but lua tables instead, so it can be similar to python dictionary or a map kind of a structure with a key value pair. One more important aspect in keymapping might about the leader key, you can set your leader key by using the global vim options with `vim.g` and access `mapleader` to set it to the key you wish. This will make the `leader` key available to us and thereafter we can map the leader key in custom mappings. ``` vim.g.mapleader = " " ``` Here, I have set my leader key to the `<Space>` key. Now, we can map keys to the existing keymaps in the vimscript. Let's map some basic keymaps first and then after setting up the plugins,we can move into plugin-specific mappings. You can also use `vim.api.nvim_set_keymap` function with the same parameters as well. ```lua vim.keymap.set('n', '<leader>w', ':w<CR>',{noremap = true}) vim.keymap.set('n', '<leader>q', ':q!<CR>',{noremap = true}) vim.keymap.set('n', '<leader>s', ':so %<CR>',{noremap = true}) vim.keymap.set('n', '<leader>ev', ':vsplit $MYVIMRC<CR>',{noremap = true}) vim.keymap.set('n', '<leader>sv', ':w<CR>:so %<CR>:q<CR>',{noremap = true}) OR vim.api.nvim_set_keymap('n', '<leader>w', ':w<CR>',{noremap = true}) vim.api.nvim_set_keymap('n', '<leader>q', ':q!<CR>',{noremap = true}) vim.api.nvim_set_keymap('n', '<leader>s', ':so %<CR>',{noremap = true}) vim.api.nvim_set_keymap('n', '<leader>ev', ':vsplit $MYVIMRC<CR>',{noremap = true}) vim.api.nvim_set_keymap('n', '<leader>sv', ':w<CR>:so %<CR>:q<CR>',{noremap = true}) ``` If, you don't like writing `vim.keymap.set` or `vim.api.nvim_set_keymap` again and again, you can create a simpler function for it. In lua a function is created just like a variable by specifying the scope of the function i.e. local followed by the `function` keyword and finally the name of the function and parenthesis. The function body is terminated by the `end` keyword. ```lua function map(mode, lhs, rhs, opts) local options = { noremap = true } if opts then options = vim.tbl_extend("force", options, opts) end vim.api.nvim_set_keymap(mode, lhs, rhs, options) end ``` Now, in this function `map`, we have passed in the same parameters like the `vim.keymap.set` function takes but we have just parsed the function in a shorter and safer way by adding `noremap = true` by default. So this is just a helper function or a verbose function for calling the vim.keymap.set function. To use this function, we can simply call `map` with the same arguments as given to the prior functions. ```lua map('n', '<leader>w', ':w<CR>') map('n', '<leader>q', ':q!<CR>') map('n', '<leader>s', ':so %<CR>') ``` Notice here, though, we have not passed the `{noremap = true}` as it will be passed by default to the `vim.api.nvim_set_keymap` or `vim.keymap.set` function via the custom map function. If you want some more examples, here are some additional mapping specific to languages, meant for compiling or running scripts with neovim instance. ```lua -- vimscript nnoremap cpp :!c++ % -o %:r && %:r<CR> nnoremap c, :!gcc % -o %:r && %:r<CR> nnoremap py :!python %<cr> nnoremap go :!go run %<cr> nnoremap sh :!bash %<CR> -- lua map('n', 'cpp' ':!c++ % -o %:r && %:r<CR>') map('n', 'c,' ':!gcc % -o %:r && %:r<CR>') map('n', 'py' ':!python %<cr>') map('n', 'go' ':!go run %<cr>') map('n', 'sh' ':!bash %<cr>') ``` So, this is how we can set up our keymaps in lua. You can customize this function as per your needs. These are just made for the understanding purpose of how to reduce the repetitive stuff in the setup. **If you are really stuck up and not feeling to convert those mappings into lua then I have a function that can do it for you, check out my dotfile repo -> [keymapper](https://github.com/Mr-Destructive/dotfiles/blob/master/nvim/lua/destructive/options.lua#L9)** ## Adding Plugin Manager Now, we really missing some plugins, aren't we? So, the neovim community has some good choices for using a new plugin manager written in core lua. It is usually a good idea to move into lua completely rather than switching to and fro between vimscript and lua. So, [Packer](https://github.com/wbthomason/packer.nvim) is the new plugin manager for Neovim in Lua, there is other plugin managers out there as well like [paq](https://github.com/savq/paq-nvim). If you don't want to switch with the plugin manager, you can still use vim-based plugin managers like [Vim-Plug](https://dev.to/vonheikemen/neovim-using-vim-plug-in-lua-3oom). So, let's install the Packer plugin manager into Neovim. We simply have to run the following command in the console and make sure the plugin manager is configured correctly. ``` # Linux git clone --depth 1 https://github.com/wbthomason/packer.nvim\ ~/.local/share/nvim/site/pack/packer/start/packer.nvim # Windows git clone https://github.com/wbthomason/packer.nvim "$env:LOCALAPPDATA vim-data\site\pack\packer\start\packer.nvim" ``` Now, if you open a new neovim instance and try to run the command `:PackerClean`, and no error pops out that means you have configured it correctly. You can move ahead to installing plugins now. Yeah! PLUG-IN time! ```lua return require('packer').startup(function() end) ``` First try to source the file, if it throws out errors it shouldn't try to fix the installation path of Packer. If the command succeded we can finally pull up some plugins. Below are some of the plugins that you can use irrespective of what language preferences you would have. This includes basic dev-icons for the status line as well as the explorer window file icons. As usual, add your plugins and make them yours. ```lua return require('packer').startup(function() use 'wbthomason/packer.nvim' use 'tpope/vim-fugitive' use { 'nvim-lualine/lualine.nvim', requires = { 'kyazdani42/nvim-web-devicons', opt = true } } use 'tiagofumo/vim-nerdtree-syntax-highlight' use 'kyazdani42/nvim-web-devicons' use 'vim-airline/vim-airline' use 'vim-airline/vim-airline-themes' end) ``` After adding the list of your plugins, you need to source the file and then install the plugins with the command `:PackerInstall`. This function will install all the plugins after the file has been sourced so make sure you don't forget it. ## Colors and Color Themes You might fancy some good-looking and aesthetic setup for neovim of course! In Neovim, we already have a wide variety of configurations to set up like color schemes, GUI colors, terminal colors, etc. You can pick up a color scheme from a large list of awesome color schemes from [GitHub](https://github.com/topics/neovim-colorscheme). After choosing the theme, plug it in the packer plugin list which we just created and source the file and finally run `:PackerInstall`. This should install the plugin. You can then set the colorscheme as per your preference, first view the color scheme temporarily on the current instance with the command `:colorscheme colorscheme_name`. ```lua return require('packer').startup(function() use 'wbthomason/packer.nvim' -- use 'Mofiqul/dracula.nvim' -- end) ``` You can then add the command to set it as the default color scheme for Neovim. ```lua vim.cmd [[silent! colorscheme dracula]] ``` You can change the background color, text color, icons style and terminal and gui style separately with the augroup and setting the colorscheme commands. ```lua vim.api.nvim_command([[ augroup ChangeBackgroudColour autocmd colorscheme * :hi normal termbg=#000030 termfg=#ffffff autocmd colorscheme * :hi Directory ctermfg=#ffffff augroup END ]]) vim.o.termguicolors = true ``` Here, I have used the background and foreground colors of the terminal variant of Neovim. Also for the Directory Explorer i.e. netrw, I have changed the terminal foreground. This you can configure as per your needs, Though this is still vimscripty, there are Autocommands and autogroups available sooner in Neovim. ## Separating Configurations If you wish to keep all your config in one file i.e. `init.lua` file, you can, though you can separate out things like `keymaps`, `plugins`, `custom_options` or if you have `lsp` configurations into separate lua packages or creating a separate module. This helps in loading up things as per requirement and also looks readable, making it a lot easier to test out things without breaking the whole config. Definitely, there will be personal preferences and pros and cons of both approaches, you can pick up whatever suits your style. ### Creating separate packages To create a separate package, we can simply add a file in the same folder as `init.vim` i.e. in the folder `~/.config/nvim` or equivalent for windows. The package name can be any valid filename with the `lua` extension. For instance, you can create a package for all your keymaps and load it in the `init.lua` as per the order you want to load them. It can be at the top, or else if you have certain base settings lower in the init file, these might not reflect or load up in the keymap package so better to load them after some of the core settings have been set. Let's create the package and dump all our maps into the keymap file package. ```lua -- ~/.config/nvim/keymap.lua function map(mode, lhs, rhs, opts) local options = { noremap = true } if opts then options = vim.tbl_extend("force", options, opts) end vim.api.nvim_set_keymap(mode, lhs, rhs, options) end map('n', '<leader>w', ':w<CR>') map('n', '<leader>q', ':q!<CR>') map('n', '<leader>s', ':so %<CR>') map('n', 'cpp' ':!c++ % -o %:r && %:r<CR>') map('n', 'c,' ':!gcc % -o %:r && %:r<CR>') map('n', 'py' ':!python %<cr>') map('n', 'go' ':!go run %<cr>') map('n', 'sh' ':!bash %<cr>') -- more keymaps ``` So, this might work if you don't have any plugin-related keymaps as it would require those functions or objects available to execute. So, we might also want to separate plugins and load them first in our keymaps or in the init file. Now, there needs to be a way for grabbing a package. Yes, there is basically like import in python or any other programming language, lua has `require` keyword for importing packages. Since the `init` file and the `keymaps` are in the same folder path, we can simply say, `require("keymap")` in our `init.lua` file. Now, it depends on your config where you want to load this package. At the top i.e. at the beginning of neovim instance or after loading the plugins down. ```lua -- init.lua require("keymaps") -- At the top -- OR -- After loading Packer plugins ``` So, now you can separate all your configs as per your requirement. It is like splitting up a puzzle and again combining them. Similar package can be created for `plugins`, `options` or `lsp` configurations. ### Creating a separate module Now, we have seen how to create a lua package and loading in neovim. We also can create modules in neovim configuration. For instance, first, the init file is loaded, Other files might not be required hence it is not loaded by default, it is only loaded when `require`ed. So, we can create a module in lua with a folder, and inside of it, we can have packages as we had in the previous method. Also, every module can have a init file loaded first when we require that module. The rest of the packages in the module are thus made available thereafter. - Module is a component not loaded by default - Only loaded when required (literally require) - Every module can have a `init.lua` file loaded first when required. - Require a package in module from outside -> `require('module_name.package_name')` So, in neovim, we need to create a `lua` folder for placing all our modules so that the lua runtime is picked up correctly. Inside this lua folder, we can create a module basically a folder. This folder name can be anything you like, I like to keep it my nickname, you can use whatever you prefer. ``` # ~/.config/nvim -- init.lua -- lua/ -- module_name/ -- init.lua -- package_name.lua -- keymaps.lua ``` Now, we can create packages in this module. You can move your keymaps package inside this folder. The keymaps package is nothing here but an example provided in the previous section for creating a package. You can basically dump all your keymaps in a single lua file and then import it from the init file. Similarly you can create a package inside a module and import it from the init file of the module(local init file `~/.config/nvim/lua/module_name/init.lua`) or the global init file(`~/.config/nvim/init.lua`). To load the packages, you have to use the same require statement irrespective of where you are importing from i.e. either from the module or from the global init file. The require statement would look like the following `require("module_name/package_name")`. Now, we can use the keymaps package to import from the module init file and then import the module from the global init file. To import a module, we can simply use the module name in the require string as `require("module_name")`. ```lua -- ~/.config/nvim -- lua/module_name/options.lua vim.opt.number = true vim.opt.tabstop = 4 vim.opt.swapfile = false -- lua/module_name/plugins.lua require("module_name.options") return require('packer').startup(function() use 'wbthomason/packer.nvim' --plugins end) -- lua/module_name/keymap.lua require("module_name.plugins") -- maps() -- lua/module_name/init.lua require("module_name.keymaps) -- init.lua require("module_name") ``` So, this is how we can create modules and packages for configurations in neovim using lua. This is also a kind of a structure for creating your own neovim plugin with lua! For further references, you can check out my [dotfiles](https://github.com/Mr-Destructive/dotfiles). ### References - [Configure Neovim for Lua](https://vonheikemen.github.io/devlog/tools/configuring-neovim-using-lua/) - [Neovim with Lua for beginners](https://alpha2phi.medium.com/neovim-for-beginners-init-lua-45ff91f741cb) - [TJ Devries Youtube](https://www.youtube.com/c/TJDeVries/videos) ## Conclusion So, that is just a basic overview of how you can get your neovim configured for lua. It is a great experience to just create such a personalized environment and play with it and have fun. You might have hopefully configured Neovim for Lua from this little guide. Maybe in the next article, I'll set up LSP in Neovim. If you have any queries or feedback, please let me know. Thank you for reading. Happy Viming :) https://www.meetgor.com/posts/golang-defer Introduction In this part of the series, we will be taking a look at the keyword in golang. The defer keyword is used for delaying the function call in a partic Wed, 29 Jun 2022 00:00:00 UTC ## Introduction In this part of the series, we will be taking a look at the `defer` keyword in golang. The defer keyword is used for delaying the function call in a particular block of program(function or a loop). ## Defer Keyword The `defer` keyword is an interesting keyword in golang, it basically holds up the execution of the statement until all the statements around the local scope has been executed. It is basically like a stack holding the execution of statements. You can have multiple defer keywords in the single code block(function or a loop), those will be called by the principle of **first in last out**. So, let's take a simple example, the syntax is quite simple just add `defer` before the statement you want to hold up. ```go package main import "fmt" func main() { fmt.Println("First") defer fmt.Println("Second Ahhh..") fmt.Println("Third") } ``` ``` go run defer.go First Third Second Ahhh.. ``` As, you can see the function call `Second` was executed at the end of all the function calls in the main function. This is because of the defer keyword. It will halt or postpone the calling of a statement/function before all the statements have bee executed in the local scope. This keyword can be stacked for calling different statements at the specific time in the state of the program. ### Multiple defer keyword We can understand the `defer` keyword in a better way if we modify the previous example a bit. We will use multiple defer statements in it for understanding the flow of the program. ```go package main import "fmt" func main() { fmt.Println("bag") defer fmt.Println("book") defer fmt.Println("cap") fmt.Println("laptop") defer fmt.Println("wallet") fmt.Println("headphones") } ``` ``` go run defer.go bag laptop headphones wallet cap book ``` Here, we can see that the `bag` is printed out first, then `laptop` and then `headphones`, but after these, we have maintained a stack for calling out the defer statements. Initially, consider a empty stack `[]` for storing defer statements. Firstly, the `bag` statement will be printed. After printing the `bag` keyword, there is a defer key word, so we add the statement to the stack `["book"]`. Just for convenience, ignore the function syntax and the actual statements, just focus on what is printed when. So, we have printed `bag` and we have stack as `["book",]`. We again encounter a `defer` keyword, so we push the statement `cap` into the stack `["book", "cap",]`. Further, we encounter the normal statement, so we print `"laptop"`. Next up, we encounter a defer keyword and thereby we push `"wallet"` into the stack, which then becomes `["book", "cap", "wallet"]`. The last statement in the main function is `"headphones"`, so we print `"headphones"` with the `Println` function. Since, there are no further statements to execute in the main function, we start poping out the function calls from the stack. Remember last in first out, so we will print the statement which were pushed last into the stack. Since the stack is `["book", "cap", "wallet"]`, so we will pop out `"wallet"` from the stack since we have pushed it last in the stack. Thereby we print `"wallet"` after `"headphones"` and the stack now becomes `["book", "cap"]`. Next, we have to pop out the second last element that we pushed into the stack which was `"cap"`, thus we print `"cap"`. The stack only has one element in the stack and we pop out that as well which is `"book"`. So, this is how the defer keyword stacks up in golang. **NOTE: The defer keyword calls functions after the execution of all other functions in it's scope but the parameters are evaluated before handed which means just the function is executed later, but the parameters are already loaded** ### defer keyword in function call The defer keyword is quite similarly used while calling the functions. The `fmt.Println` is also a function, but writing custom functions gives a different feeling to us. ```go package main import "fmt" func square(x int) int { fmt.Println(x * x) return x * x } func main() { // defer keyword in function defer square(12) defer square(10) square(15) } ``` ``` go run defer.go 225 100 144 ``` So, we have created the `square` function and called it thrice and twice we have used the `defer` keyword. The defer keyword first pushes the function call `square(12)` to the stack, so the stack is `[square(12) ]`. Next, we again have the defer keyword, so we push the `square(10)` to the stack which becomes `[square(12) square(10)]`. The next statement is immediately called the function `square(15)` and thereby we end the main function. So, we have to pop the function calls from the stack. The last element `square(10)` is called first and then the only call to the `square(12)`. Thereby, the order of the calling becomes `225 100 144`. That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/mr-destructive/100-days-of-golang/tree/main/scripts/defer/defer.go) GitHub repository. ## Conclusion So, from this post, we were able to understand the defer keyword in golang. We were able to explore how we can delay the calling of a function in a particular scope of the program. Thank you for reading, if you have any queries, feedback, or questions, you can freely ask me on my social handles. Happy Coding :) https://www.meetgor.com/posts/golang-strings Introduction In the 15th post of the Series, we will be looking into the details of the String manipulation and performing types of operations in Golang. We wil Tue, 28 Jun 2022 00:00:00 UTC ## Introduction In the 15th post of the Series, we will be looking into the details of the String manipulation and performing types of operations in Golang. We will explore string manipulation, concatenation, helper functions, etc. which will help in working with strings in Golang. ## String Concatenation String Concatenation refers to the combining and formatting of strings in Golang. We can combine multiple strings and formating the way we display the strings in Golang. We have a few ways and functions to concatenate strings in Golang. ### Using the + operator We can simply concatenate strings using the `+` operator, though keep in mind you should only concatenate the string with a string and not any other data type like integer, or float, it will throw out errors for mismatched string types. ```go package main import ( "fmt" ) func main() { s1 := "Go" s2 := "Programming" s3 := "Language" s := s1 + s2 + s3 fmt.Println(s) } ``` ``` $ go run concatenate.go GoProgrammingLanguage ``` The `+` operator will literally join the strings as it is and form a string. ### Using the += operator The other way to continuously append a string to an existing string, we can use the `+=` operator. This operator will append the provided string to the end of the original string. ```go p := "Story" p += "Book" fmt.Println(p) ``` ``` go run concatenate.go StoryBook ``` ### Using the Join method The join method is a function available in the string package in Golang. We can join strings elements in a slice or an array using the [Join](https://pkg.go.dev/strings#Join) method in the strings [package](https://pkg.go.dev/strings) in golang. The Join method will combine all the elements in between the elements with a particular string. So, the function takes two parameters `Join(array, string)`, the array or a slice is parsed into the function which will be used to insert the provided string in between the elements of the slice. #### Join - Parameters : array/slice, string - Return Value : string ```go q := []string{"meetgor.com", "tags", "golang", "string"} r := strings.Join(q, "/") fmt.Println(r) ``` ``` go run concatenate.go meetgor.com/tags/golang/string ``` In the above example, we use have used the `Join` method to insert a string in between the elements of a slice of strings. The string `"/"` has been inserted in between the elements, and the elements are combined as a single string. So, each individual element starting from the `0` index `meetgor.com` is appended the string `/` and further the next element `tags` have been appended and the procedure caries on till the last element. Note that the string is not inserted after the last element. The function `Join` returns a string and thereby we store the string in a variable. ### Using Sprintf method We can use the [Sprintf](https://pkg.go.dev/fmt#Sprintf) function from the fmt package to format the string by storing the string rather than printing it to the console. The sprintf function is quite similar to the `Printf` but it only parses strings rather than printing them directly to the console. ```go // Using Sprintf function to format strings name := "peter" domain := "telecom" service := "ceo" email := fmt.Sprintf("%s.%s@%s.com", service, name, domain) fmt.Println(email) ``` ``` go run concatenate.go ceo.peter@telecom.com ``` The sprintf function basically allows us to concatenate strings in a defined format just like we use `printf` to print formatted strings. In the above example, we have formatted three strings in the form of an email by assigning a placeholder for the string i.e. `%s`, and adding the required characters in the formatted string. ### Using Go string Builder method The [Builder](https://pkg.go.dev/strings#Builder) type is provided by the strings package in Golang. The Builder type helps in building strings in an efficient way. By creating a string builder object, we can perform operations on a String. ```go package main import ( "fmt" "strings" ) func main() { // Using Builder function c := []string{"j", "a", "v", "a"} var builder strings.Builder for _, item := range c { builder.WriteString(item) } fmt.Println("builder = ", builder.String()) b := []byte{'s', 'c', 'r', 'i', 'p', 't'} for _, item := range b { builder.WriteByte(item) } fmt.Println("builder = ", builder.String()) builder.WriteRune('s') fmt.Println("builder = ", builder.String()) fmt.Println("builder = ", builder) } ``` ``` go run concatenate.go builder = java builder = javascript builder = javascripts builder = {0xc000088dd8 [106 97 118 97 115 99 114 105 112 116 115]} ``` The builder structure provided by the strings package is quite important for working with strings in an efficient manner. Its usually used for string concatenation operations. We can perform write operations to the buffer which is a byte slice. Here we have created the `builder` variable which is of type `strings.Builder`, further we have appended the string to it in a for a loop. So, we construct a string from the string list elements, they can be even rune slice or byte slice. We have used three methods here, the `WriteString`, `WriteByte`, and `WriteRune` which are quite obliviously used for writing `string`, `byte`, and `rune`to the string builder object. ### Using the Bytes buffer method The `bytes` package also has something similar to `Builder` type in `string` as [Buffer](https://pkg.go.dev/bytes#Buffer.Bytes). It has almost the same set of methods and properties. The main difference is the efficiency, `strings.Builder` is comparatively faster than the `bytes.Buffer` type due to several low-level implementations. We can discuss those fine details in a separate article but right now we'll focus on the ways we can utilize this type for string concatenation. ```go package main import ( "fmt" "bytes" ) func main() { // Using bytes buffer method var buf bytes.Buffer for i := 0; i < 2; i++ { buf.WriteString("ja") } fmt.Println(buf.String()) buf.WriteByte('r') fmt.Println(buf.String()) k := []rune{'s', 'c', 'r', 'i', 'p', 't'} for _, item := range k { buf.WriteRune(item) } fmt.Println(buf.String()) } ``` ``` go run concatenate.go jaja jajar jajarscript {[106 97 106 97 114 115 99 114 105 112 116] 0 0} ``` So, like for the `strings.Builder` type, we have [WriteString](https://pkg.go.dev/bytes#Buffer.WriteString), [WriteByte](https://pkg.go.dev/bytes#Buffer.WriteByte), and [WriteRune](https://pkg.go.dev/bytes#Buffer.WriteRune) in the `bytes.Buffer` type. We can use it exactly the way we do with the previous example. Also, the `bytes.Buffer` type returns a slice of bytes so we will have to use the [String()](https://pkg.go.dev/strings#Builder.String) method to format it as a string. If we look at the [bytes.Buffer](https://go.dev/src/bytes/buffer.go) type, it returns a slice of bytes and two more properties viz. `off` and `lastRead`. These two properties are used for indicating the index of the byte in the buffer and reallocation of the buffer. This is too low-level stuff that can be explored and explained in a separate section. For further readings on the bytes Buffer or String Builder types, you can follow up with these articles: - [bytes.Buffer vs strings.Builder](https://medium.com/@felipedutratine/string-concatenation-in-golang-since-1-10-bytes-buffer-vs-strings-builder-2b3081848c45) - [Bytes Buffer](https://syslog.ravelin.com/bytes-buffer-i-thought-you-were-my-friend-4148fd001229) - [Best ways to use bytes.Buffer](https://stackoverflow.com/a/49295215) ## String Comparison Now, we can move into the comparison of Strings in Golang. We have quite a few ways to compare strings in golang. We cover some of them in this section. ### Using Comparison operators The basic comparison can be done with the comparison operators provided by Golang. Just like we compare numeric data we can compare strings. Though the factor with which we compare them is different. We compare them by the lexical order of the string characters. ```go package main import "fmt" func main() { s1 := "gopher" s2 := "Gopher" s3 := "gopher" isEqual := s1 == s2 //"gopher" is not same as "Gopher" and hence `false` fmt.Printf("S1 and S2 are Equal? %t ", isEqual) fmt.Println(s1 == s2) // "gopher" is not equal to "Gopher" and hence `true` fmt.Println(s1 != s2) // "Gopher" comes first lexicographically than "gopher" so return true // G -> 71 in ASCII and g -> 103 fmt.Println(s2 < s3) fmt.Println(s2 <= s3) // "Gopher" is not greater than "gopher" as `G` comes first in ASCII table // So G value is less than g i.e. 71 > 103 which is false fmt.Println(s2 > s3) fmt.Println(s2 >= s3) } ``` ``` go run comparison.go S1 and S2 are Equal? false false true true true false false ``` In the above examples, we are able to see the comparison of two strings. There are three strings, two of which are identical, and the third is identical as well but is not equal considering the case of the characters in the string. We have compared the strings in order of the ASCII value of the characters of the strings. For example, A (65) comes before a (97). Similarly, numbers come before letters. So accordingly the comparison of these string characters decides the result. For the ASCII table, you can take a look over the below image:  ### Using Compare method We also have the [Compare](https://res.cloudinary.com/techstructive-blog/image/upload/v1656423519/blog-media/ascii_table.png) method in the strings package for comparing two strings. The comparison method returns an integer value of either -1, 0, or 1. If the two strings are equal, it will return 0. Else if the first string is lexicographically smaller than the second string, it will return -1, else it will return +1. #### strings.Compare - Return Type: Int (-1, 0, 1) - Parameters: string, string You can check out the [source code](https://go.dev/src/strings/compare.go) for further clarity. ```go package main import( "fmt" "strings" ) func main() { s1 := "gopher" s2 := "Gopher" s3 := "gopher" fmt.Println(strings.Compare(s1, s2)) fmt.Println(strings.Compare(s1, s3)) fmt.Println(strings.Compare(s2, s3)) } ``` ``` go run comparison.go 1 0 -1 ``` In the above example, the two strings `s1` and `s2` are compared and it returns the integer value `+1`, indicating the first string is lexicographically greater than the second string which is true `"gopher"` will be lexicographically after `"Gopher"` due to the presence of `G`. In the second example, we are comparing the strings `s1` and `s3` which are equal, and hence the function returns `0` as expected. In the third example, we are comparing the strings `s2` and `s3` identical to the first case but here order matters. We are comparing `"Gopher"` with `"gopher"` so the first string is lexicographically smaller than the second string and thereby returning `-1` as discussed. ### Using strings EqualFold We also have another method in the strings library called [EqualFold](https://pkg.go.dev/strings#EqualFold) which compares two strings lexicographically but without considering the case precedence. That is the upper case or lower case is ignored and considered equal. So we are truly case-insensitively comparing the strings. #### strings.EqualFold - Return Type: bool (true or false) - Parameters: string, string ```go package main import( "fmt" "strings" ) func main() { s1 := "gopher" s2 := "Gopher" s3 := "gophy" fmt.Println(strings.EqualFold(s1, s2)) fmt.Println(strings.EqualFold(s1, s3)) fmt.Println(strings.EqualFold(s2, s3)) } ``` ``` go run comparison.go true false false ``` So, in the above example, we are comparing the strings `"gopher"` and `"Gopher"` i.e. `s1` and `s2`, which are equal if we think case-insensitively. Hence the method returns true, they are equal. In the next example, we compare the strings, `s1` and `s3` i.e. `"gopher"` `"gophy"` which are not equal, and hence we return `false`. Similar is the case for `"Gopher"` and `"gophy"` which is `false`. Also, if we consider two strings `"gophy"` and `"gophy"` it will quite obliviously return `true`. ## String Manipulation and utility methods The strings package in golang has some great utility methods for working with string or any form of text. We will explore some of the quite useful and widely used utilities in the strings package. ### ToLower, ToUpper and Title functions The strings package also provides some utility functions for operating on the case of the characters in the strings. We have functions like [ToLower](https://pkg.go.dev/strings#ToLower), [ToUpper](https://pkg.go.dev/strings#ToUpper), and [Title](https://pkg.go.dev/golang.org/x/text/cases#Title) which can be used to convert the string into lower case, uppercased or Capitalised(Title) cased characters respectively. #### strings.ToLower - Return Type: string - Parameters: string #### strings.ToUpper - Return Type: string - Parameters: string #### cases.Title - Return Type: Caser - Parameters: Language Options ```go package main import ( "fmt" "strings" "golang.org/x/text/cases" "golang.org/x/text/language" ) func main() { s1 := "Ubuntu 22" s2 := "meet" s3 := "IND" fmt.Println(strings.ToLower(s1)) fmt.Println(strings.ToLower(s2)) fmt.Println(strings.ToLower(s3)) fmt.Printf(" ") fmt.Println(strings.ToUpper(s1)) fmt.Println(strings.ToUpper(s2)) fmt.Println(strings.ToUpper(s3)) fmt.Printf(" ") cases := cases.Title(language.English) fmt.Println(cases.String(s1)) fmt.Println(cases.String(s2)) fmt.Println(cases.String(s3)) } ``` ``` # 100-days-of-golang/scripts/strings go mod init go get golang.org/x/text/cases go get golang.org/x/text/language go run utility.go ubuntu 22 meet ind UBUNTU 22 MEET IND Ubuntu 22 Meet Ind ``` Here, we can see that the function, `ToLower` has converted all the characters of a string to the lower case of the alphabet. Similarly, the `ToUpper` function has turned the characters of the strings to their respective alphabetical upper case. The [Title](https://pkg.go.dev/strings#Title) method in the strings package has been deprecated due to incompatibility with certain languages and cases. So, we are using the [text/cases](https://pkg.go.dev/golang.org/x/text/cases) package to get the [Title](https://pkg.go.dev/strings#Title) method that appropriately converts a string to Title cased. To set up this function, you need to perform a certain package installation process which is quite straightforward. Just create a go mod which is used for managing dependencies and packages for a project. So run the commands given below in the same order in your local setup: ``` go mod init go get golang.org/x/text/cases go get golang.org/x/text/language ``` This will set up a go.mod file and install the packages namely the `cases` and `language` packages. After doing this you will be able to access the functions `Title` from the cases package which can be imported by the format `"golang.org/x/text/cases"` and `"golang.org/x/text/language"`. Now, we can use the Title function and parse the parameters which is the language type. Here we have used the `language.English` which is a [language Tag](https://pkg.go.dev/golang.org/x/text@v0.3.7/language#Tag) to say use the semantics of English language while parsing the title cased characters. We now assign the value of the function `Title` to a variable as it will be of type `Caseer` and we want to still parse the string into the function. The [caser](https://pkg.go.dev/golang.org/x/text/cases#Caser) object will have certain methods and properties attached to it, we will use the method [Strings](https://pkg.go.dev/golang.org/x/text/cases#Caser.String) that will convert the given string into the title cased string. Hence we return the title cased string using the title function with the help of cases and language packages. ### Contains and ContainsAny functions In the strings package, we have the [Contains](https://pkg.go.dev/strings#Contains) and [ContainsAny](https://pkg.go.dev/strings#ContainsAny) method which checks for the presence of substrings within a string. This will help in looking up hidden patterns and find for substrings in a string. #### strings.Contains - Parameters: string, string - Return Type: bool (true, false) The Contains method helps in getting the exact match of the substring in the given string. Firstly, the method takes two parameters one being the actual string and the other being the substring that you want to find inside the string. Let's say we have the `string="bootcamp"` and `substr="camp"`, then the `Contains` method will return true because the string contains the substring `camp`. #### strings.ContainsAny - Parameters: string, string - Return Type: bool (true, false) The `ContainsAny` method just like the Contains method takes two parameters string and the other as the substring, but it would return true if it finds any character or a single byte(Unicode chars) inside the string. Let's say the `string="google photos"` and `substring="abcde"`, then the `ContainsAny` method will return true because the string contains at least one character in the substring which is `e`. ```go package main import ( "fmt" "strings" ) func main() { str := "javascript" substr := "script" s := "python" fmt.Println(strings.Contains(str, substr)) fmt.Println(strings.Contains(str, s)) fmt.Println(strings.ContainsAny(str, "joke")) fmt.Println(strings.ContainsAny(str, "xyz")) fmt.Println(strings.ContainsAny(str, "")) } ``` Here, we have used the string methods like `Contains` and `ContainsAny` to find the substring inside a string. In the first example, the `str` variable is assigned as `"javascript"` and `substr` string as `"script"`. We use the `Contains` function with parameters `(str, substr)`. This will return `true` as we can see the `"script"` is a substring of `"javascript"`. Also, we have initialized the variable `s` to `"python"`. We use the `Contains` method with the parameters `(str, s)` which will return `false` as `"python"` is not a substring of `"javascript"`. The next set of examples is of the `ContainsAny` method which will return true for any character present in the substring is present in the string :). Quite a Simple right, Just understand that any character in your substring present in the string will return `true`. As we see in the example, `ContainsAny` method is used with the parameters `("javascript", "joke")`, It will return `true` as `j` is present inside the string, though the entire substring is not present. The next example is by passing `("javascript", "xyz")` to the method `ContainsAny` will return `false` as we don't have either `"x"`, `"y"`, or `"z"` in the string. So this is how the `ContainsAny` method works. Other similar methods you might be interested in learning are: [Index](https://pkg.go.dev/strings#Index), [IndexAny](https://pkg.go.dev/strings#IndexAny), [LastIndex](https://pkg.go.dev/strings#LastIndex), etc. you can find the list of methods in the [strings package](https://pkg.go.dev/strings#pkg-functions). ### Split and SplitAfter functions We also have methods to manipulate the string for splitting the characters or bytes with certain patterns. In the strings package, the [Split](https://pkg.go.dev/strings#Split) and [SplitAfter](https://pkg.go.dev/strings#SplitAfter) methods are quite handy to know about. ```go package main import ( "fmt" "strings" ) func main() { // Split method for splitting string into slice of string link := "meetgor.com/blog/golang/strings" fmt.Println(strings.Split(link, "/")) fmt.Println(strings.SplitAfter(link, "/")) // SplitAfter method for splitting string into slice of string with the pattern data := "200kms50kms120kms" fmt.Println(strings.Split(data, "kms")) fmt.Println(strings.SplitAfter(data, "kms")) } ``` ``` go run utility.go [meetgor.com blog golang strings] [meetgor.com/ blog/ golang/ strings] [200 50 120 ] [200kms 50kms 120kms ] ``` So, from the above examples, we can see how `Split` and `SplitAfter` methods work. The Split method splits the text before and after the pattern string also removing the pattern string whereas the `SplitAfter` method keeps the pattern and splits it after it, hence we see the pattern string getting attached to the left string. In the first example, we see the `link` variable being split into strings as `"/"` being the separator. The Split method returns the slice of the string elements which have been split. In the `data` variable, the `"kms"` is the separator, so we get the resultant slice as `[200, 50, 120]`, the `"kms"` string acts as a separator and it is ignored. In the next example, we see the `link` variable being split into strings as `"/"` being the separator as previously but here, the splitting occurs after the separator has been parsed. So, we see `"meetgor/"` being split after the separator string and then `"blog/"` and so on. Also the next example, in the `data` variable, we see `"200kms"` being split instead of `200` using Split. #### Repeat and Fields functions In the strings package, we have methods like [Repeat](https://pkg.go.dev/strings#Repeat) and [Fields](https://pkg.go.dev/strings#Fields) for manipulating the text inside the string. These methods are used to populate or extract data from the string. ##### strings.Repeat - Parameters: string, int - Return Type: string The `Repeat` method is used to create a string by repeating it n number of times and appending it to the string which is returned as the final string. So, the `Repeat` method takes in two parameters the string to repeat, an integer as a count to repeat the string, and returns a string. ```go package main import ( "fmt" "strings" ) func main() { // Repeat method for creating strings with given string and integer pattern := "OK" fmt.Println(strings.Repeat(pattern, 3)) } ``` ``` go run utility.go OKOKOK ``` So in this example, we can see that the string `"OK"` is passed to the method `Repeat` with the integer `3` and thus it is appended into itself three times and the resultant string becomes `"OKOKOK"`. ##### strings.Fields - Parameters: string - Return Type: []string The `Fields` method is used to extract the words from the string, that is the characters/bytes and group them with one or more consecutive white spaces. The function returns a slice of string. ```go package main import ( "fmt" "strings" ) func main() { // Fields method for extracting string from the given string with white space as delimiters text := "Python is a prgramming language. Go is not" text_data := strings.Fields(text) fmt.Println(text_data) for _, d := range text_data { fmt.Println("data = ", d) } } ``` ``` go run utility.go [Python is a prgramming language. Go is not] data = Python data = is data = a data = prgramming data = language. data = Go data = is data = not ``` The above example demonstrates the usage of `Fields` which will extract characters and split after encountering whitespace. So, we return a slice of string in which the string elements are split before white space. Thus, using the Fields method we get the words or characters as space-separated values in the slice. You can even expand on this method with the [FieldsFunc](https://pkg.go.dev/strings#FieldsFunc) method which allows you to write a custom delimiter option and extract data according to your requirement. There are tons of methods in the strings package for working with strings, you can see a detailed list of functions in the [documentation](https://pkg.go.dev/strings#pkg-functions). That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/mr-destructive/100-days-of-golang/) GitHub repository. ## Conclusion So, from this post, we were able to understand the different methods and types to concatenate and interpolate strings in golang. We explored different types of concatenating strings, string comparison, and various methods for manipulating and interpolating strings. Thank you for reading, if you have any queries, feedback, or questions, you can freely ask me on my social handles. Happy Coding :) https://www.meetgor.com/posts/golang-mutable-immutable Introduction In this 14th Post of the 100 days of GOlang, we will be understanding about the mutable and immutable data types in Golang. Firstly, we will unders Tue, 21 Jun 2022 00:00:00 UTC ## Introduction In this 14th Post of the 100 days of GOlang, we will be understanding about the mutable and immutable data types in Golang. Firstly, we will understand the concept of mutability and understand the differences in mutable and immutable data types, further we will explore which data types in Golang are Mutable and Immutable. ## Mutable Data Type Mutable data type is a data type which can be modified without reallocating any chunk of the memory assigned at the time of initialization. In simple words, a variable is mutable if its value can be altered without reallocating itself to a new memory space. In mutable data type the value located in a memory address can be modified. This means we do not have to reallocate any memory or change the pointer of a variable to point to other address in order to change the value of the variable. ``` // Initialization int age = 12 // The value of age is 12 which is stored at some memory address let's sayy 0x12345 // Modifying the value int age = 13 // If the data type is mutable, we can directly change the value in the memory address ``` Mutable data type looks quite simple by using a single variable to demonstrate, but if we scale the example a little, things start to escalate pretty quickly. ``` // Initialization int age = 12 // The value of age is 12 which is stored at some memory address let's sayy 0x12345 int experience = age // This will make the experience variable point to the address which the age variable is pointing to. // Yes, It will point to the same memory address // Variable | Memory Address | Value // age -> 0x12345 -> 12 // experience -> 0x12345 -> 12 // Modifying the value int experience = 17 // A change in either of the variable will change the value in both the variables // Since both the variables are poiting to the same memory address // Variable | Memory Address | Value // age -> 0x12345 -> 17 // experience -> 0x12345 -> 17 // If the data type is mutable, we can directly change the value in the memory address ``` ## Immutable Data Types Immutable data type is a data type which cannot be modified without allocating a new memory. So, the immutable data type has to reallocate memory for making changes to the value of a variable. This might be a downside if the variable is holding a large sets of values, it will require a lot of memory re-allocation for a slight change in the value. Immutable data types also mean that you cannot change the value in the memory address which the variable is pointing to, but you can make the variable point to a different memory location under the hood to change or modify the content of a variable. ``` // Initialization string fruit = "apple" // the value "apple" is stored in a memory location let's say 0x12345 originally // Modifying the value string fruit = "orange" // the value "orange" will be located in a different memory location say 0x98765 // The memory address that the variable fruit points to is changed and not the value of the memory address // This is called immutablility in data types ``` This is the basics of mutability and immutability, this might be a bit difficult to digest, but take your time and understand it thoroughly. This concept is critical for understanding the under the hood point of view while debugging in several occasions. ## Mutable data type in Golang In golang there are a few mutable data types 1. Slice 2. Array 3. Map 4. Channels ### Slice and Arrays Slice and Arrays are mutable data types in golang, this means the value of the elements in slice or array can be changed after initialization without re-allocations of memory. ``` go package main import "fmt" func main() { // Slice s := []int{1, 2, 3} fmt.Printf("S[1] -> %p ", &s[1]) p := s p[1] = 4 fmt.Printf("S[1] -> %p ", &s[1]) fmt.Println("s =",s) fmt.Println("p =",p) } ``` ``` go run mutable.go S[1] -> 0xc000018200 S[1] -> 0xc000018200 s = [1 4 3] p = [1 4 3] ``` In the above example, we can see that the slice has been initialized to `{1,2,3}` another slice has been referenced to that string i.e. it will point to the same address as the original string `s`. If we modify the slice `p`, since it is pointing to the same memory address as the slice `s` is pointing it will modify the slice `s` as well (they are the same slice). This is what mutability does, it modifies the value stored in a memory address directly, without allocating any extra memory to the variable. ### Arrays You won't be able to see the concept of mutable data types with arrays as they are not referenced to any memory address, it is a collection of a single type of value and it is thus static. Since it is not a reference to any memory address, the value of the elements doesn't change if we change the value of an element in the copy of the array. ```go package main import "fmt" func main() { // Arrays a := [3]int{10, 20, 30} fmt.Printf("A[1] -> %p ", &a[1]) b := a b[1] = 40 fmt.Printf("A[1] -> %p ", &a[1]) fmt.Println("a =", a) fmt.Println("b =", b) } ``` ``` go run mutable.go A[1] -> 0xc0000aa038 A[1] -> 0xc0000aa038 a = [10 20 30] b = [10 40 30] ``` In the above example, we can see the array `a` is initialized with a fixed length and initialized values. We then create another array by assigning the array `a` to it, this creates a copy of a collection of the elements in a different memory location. So, if we change an element in the array `b`, there won't be any change in the elements of array `a` as the elements are stored in a completely different memory location. ### Map Map is similar to slices in a way they are references to the memory address. A map as we have explored in the [seventh part](https://www.meetgor.com/golang-maps) of the series], they are a pair of key and value pairs. The map is internally a reference to a hash map, a hash map is an abstract data type or a structure in Golang, it basically is an array of buckets. Buckets contain high-order bits with a hash(random value) to make the keys distinct in the map. The number of buckets is initially 8, but it expands as required so it doubles the number of buckets and assigns the value to the map elements. For a detailed reference, you can look at the source implementation of [golang's map](https://github.com/golang/go/blob/master/src/runtime/map.go). So, a map is mutable, which means if we change the value of a key it is changed directly to the memory rather than reallocating the memory space for the entire map. ```go package main import "fmt" func main() { // Map m := map[string]int{"level": 5, "health": 9} fmt.Println("m =", m) n := m n["food"] = 12 fmt.Println("m =", m) fmt.Println("n =", n) } ``` ``` go run mutable.go m = map[health:9 level:5] m = map[food:12 health:9 level:5] n = map[food:12 health:9 level:5] ``` In the above example, we can see the map `m` is a string integer map with 2 keys. We create a new map called `n` and reference it as the map `m` this will make the map `n` point to the same hash map as the map `m` does. Hence, if we change the value of a key in the map `n` it will thereby change the map `m` as maps `m` and `n` both point to the same map (in memory address). ## Immutable data type in Golang In golang, there are a few immutable data types as well like string, pointers, boolean, and core data types like integer, float, etc. As we discussed immutable data types, are data types that don't change the value of the variable directly into the provided memory address, it re-allocates the memory address with the new value/edited value. 1. Boolean, Int, Float 3. Pointers 3. String 4. Interfaces ### Boolean The boolean data type on golang is an immutable data type which means it re-allocates the memory for any change in the value of the boolean variable. Boolean variables are simple as they can have two values either `true` or `false`. If we declare a boolean variable initialize it with a value, if we further wanted to change the value of the variable, it is done by reallocating the memory address which was initially holding the value. ```go package main import "fmt" func main() { // bool boolean := true b := boolean b = false fmt.Println("boolean = ", boolean) fmt.Println("b = ", b) } ``` ``` go run immutable.go boolean = true b = false ``` In this example, we can see that the boolean variable `boolean` (I have literally named it boolean) is initialized as `true`, next we declare another variable `b` and set it to the value of `boolean` which is the value `true`. We then change the value of the variable `b` but the value of `boolean` does not change. It doesn't give much clarity on the immutability of the boolean data type. This is because it is an under an hood process and a low-level process in programming in golang. Immutable doesn't mean you cannot change values, but the value is not directly changed, there happens some under-the-hood (low level) stuff to alter the value and change the memory address which is ben pointed as the location which holds the value. To get more clarity, you can refer to this [Stack-Overflow discussion](https://stackoverflow.com/questions/71589811/go-ints-and-strings-are-immutable-or-mutable/71590289#71590289). ### Pointers Pointer as well is an immutable data type in golang, we cannot change the value in this case the memory address of a variable which we are pointing to, directly but we need to re-allocate the memory for changing the value of the variable. ```go package main import "fmt" func main() { // Pointer n := 567 t := 123 ptr := &n ptr_new := ptr fmt.Println("ptr = ", ptr) fmt.Println("ptr_new = ", ptr_new) ptr_new = &t fmt.Println("ptr = ", ptr) fmt.Println("ptr_new = ", ptr_new) } ``` ``` go run immutable.go ptr = 0xc0000aa008 ptr_new = 0xc0000aa008 ptr = 0xc0000aa008 ptr_new = 0xc0000aa020 ``` Clearly, in the example, we have initially created a pointer `ptr` which points to the variable `n`, then we create another pointer `ptr_new` and assign it to the value of the pointer `ptr`. This will make both the pointers point to the same variable `n`. We then point the `ptr_new` pointer to the variable `t`, this changes the value of pointer `ptr_new` but since the pointer stores a memory address, it was not holding the actual value at the memory address. So, the change in the pointing value of `ptr_new` does not change the value of the pointer `ptr`. Again, immutable types cannot be seen actually changing the memory location of those variables, it is just the low-level implementation that sometimes needs to be kept in mind. ### String Strings are the classical example of immutable data types in golang, this data type is quite commonly used and is quite important for creating a wide variety of applications. The value of the string variable can be changed but the process happens with/without changing the value of the memory address of the initial value, we have to change the memory address of the variable in order to change the value. This might not be evident by even using `&variable_name` as it doesn't give the actual insight about the internal working of how the memory address might be processed at run time. ```go package main import "fmt" func main() { // String str := "gopher" str_copy := str str_copy = "cooper" fmt.Println("str = ", str) fmt.Println("str_copy = ", str_copy) } ``` ``` go run immutable.go str = gopher str_copy = cooper ``` Here, in the above example, we have created a variable `str` with a value `"gopher"` and then, another variable `str_copy` which is assigned the value of `str`. If we change the value of `str_copy`, the value of `str` is not changed, this is not giving any information on the immutability of a data type. What actually gives a clear understanding of the immutability of data type in the string is the modification of the character of a string. We cannot change the character at a particular index of the string in golang. ```go package main import "fmt" func main() { // Character at index cannot be changed in string s := "StarWars" s[4] = 'C' // s[4] = "C" // also won't work fmt.Println(s) } ``` ``` go run immutable.go immutable.go:18:2: cannot assign to s[4] (value of type byte) ``` So, we can see we cannot change the value of the internal characters of the string. This is why string data type is immutable, we cannot change the value of internal elements of the variable directly but we can change the value of the variable as a whole as we saw in the previous example. So this is how the immutable data types are perceived in golang, we cannot change the value directly at the memory address, we change the internal location for any change in the value which is dependent on the decision by the garbage collector. That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/Mr-Destructive/100-days-of-golang/tree/main/scripts/im-mutable) GitHub repository. ## Conclusion So, from this small post, we were able to understand the different behaviors of data types in golang. We can understand it at the beginning of the variables part of the series, but it becomes more clear when we play with all the data types and then understand how and why they behave as they do. This was not a deep dive into the core working of immutable and mutable data types but gave a fair bit of understanding of the behavior of different data types in golang. Thank you for reading, if you have any queries, feedback, or questions, you can freely ask me on my social handles. Happy Coding :) https://www.meetgor.com/posts/django-deploy-railway Introduction We have already seen the process of deployment of a Django app on Heroku, now we can move to another platform which is . This allows a faster and m Thu, 19 May 2022 00:00:00 UTC ## Introduction We have already seen the process of deployment of a Django app on Heroku, now we can move to another platform which is [Railway App](https://railway.app). This allows a faster and more intuitive way to deploy a Django app(or any sort of web app). This platform has several [Starter Apps](https://railway.app/starters) for shipping your app in several clicks, definitely, it would require some configuration and tweaking a bit. We will be following similar steps from the [Django-Heroku Deployment Guide](https://mr-destructive.github.io/techstructive-blog/series/django-deployment) for configuring and setting up our Django project for the deployment on the Railway app. We explore a few different ways to create and deploy a Django project on Railway. ## Creating a Project on Railway App Once we have our Django project setup, we can move ahead and create a Railway Project. A Railway project is a collection of environments, plugins, services, and deployment variables. By heading on the [New Project](https://railway.app/new) webpage, you can log in or sign up for a free account and create a simple project. Here we can have a few options: 1. Create a Django App Template (Djangopy Starter) 2. Pull a Project from a GitHub Repository (Deploy from GitHub Repo) We'll look into each of them, the first one is by allowing the `Railway` bot to create a prebuilt Django template on our GitHub account. The second option is for fetching and deploying an existing Django project repository on GitHub. ## Railway Django Project Template Railway provides a simple Django project template, it consists of a single app and a simple view that displays a `HttpResponse`. The Django project template provided by Railway is open source and available on [GitHub](https://github.com/railwayapp/starters/tree/master/examples/djangopy). With this method, you won't require any django project to create by yourself, it would be a template of django project created by the railway bot. Though for extending and building a meaningful django project, you will have to clone the repository and make necessary changes to it. Create a Project by heading on to the [New Project](https://railway.app/new) Tab and search for `Django`, you should see a `django` project as a starter pack.  Once you select the Django project, we have an initial configuration tab opened up for us. It is mandatory to fill the `SECRET_KEY` environment variable as it makes your django project more secure. Also, do change the name of the `repository` as it will be the name of your repository name in your GitHub account. To create a `SECRET_KEY` key, you can move into your terminal and do some python wizardry. Open a Python REPL, by entering the command `python` or `python3`. Import the `secrets` module and run the function `token_hex(24)` which will generate a key of length 24. ``` python import secrets secrets.token_hex(24) ```  Now, copy the `SECRET_KEY` without quotes into the prompt and this will create a repository on your GitHub with the provided name. The `Railway` Bot will create a django project with the name `djangopy` in that repository with some pre-configured settings.   This will create a simple repository on your GitHub but also a django application deployed on Railway along with PostgreSQL Database attached to it.  So, this has deployed the project on Railway with this [https://djangopy-production-43cb.up.railway.app/](https://djangopy-production-43cb.up.railway.app/) URL Link. The name of the link can be configured from the Settings Tab in Dashboard Section and editing the Service Domains and even adding a Custom domain. So this is how we deploy a basic django application on railway app. Further, you can modify the contents of the Github repository and push the code by committing the changes and it will pick it from there and also deploy it automatically, thereby creating a sort of CI-CD. We'll be demonstrating this in the next section which is a method to deploy the project from a custom GitHub repository i.e. by setting up everything ourselves and then attaching the Railway app for deployment. The benefit of this method is that we can configure some already existing Django applications by tinkering a bit. ## Deploying from GitHub repository to Railway This is the second method for deploying the Railway project. For this we need a proper Django Project, we will set up a django application from scratch, I have already created a simple CRUD application in Django for a Blog on [GitHub](https://github.com/Mr-Destructive/django-blog). This won't be a Django guide for setting up views and creating models, I've explained all the setup of the django-blog in my [Django Basics series](https://mr-destructive.github.io/techstructive-blog/series/django-basics). ### Creating a Django Application For deploying an app, we definitely need an app, we need to create a basic Django application to deploy on the web. We'll be creating a simple blog application with a couple of views and a simple model structure. As for the database, we'll be using Postgres as Railway has an database service for it and it is pretty easy to configure. #### Set up a virtual environment We need to set up a virtual environment in order to keep the Django project neat and tidy by managing the project-specific dependencies and packages. We can use the [virtualenv](https://pypi.org/project/virtualenv/) package to isolate a python project from the rest of the system. ``` # install the virtualenv package pip install virtualenv # create a virtual env for the project virtualenv .venv # activate the virtualenv Windows: .venv\Scripts�ctivate Linux/macOS: source .venv/bin/activate ``` This will set up the project nicely for a Django project, you now install the core Django package and get started with creating a Django application. ```bash # install django pip install django # start a django project django-admin startproject blog . cd blog # create a application in django project python manage.py createapp api # Create some models, views, URLs, templates # run the server python manage.py runserver ``` We assume you already have a Django project configured with some basic URLs, views, and templates or static files as per your project and requirements, for this tutorial I will be using the simple blog application from my previous Django tutorials as a reference. As said earlier, you can follow along with my [Django Basics](https://techstructiveblog.hashnode.dev/series/django-basics) series and refer to the Blog Application project on [GitHub](https://github.com/Mr-Destructive/django-blog). ### Configuring the Django Application Make sure to create and activate the virtual environment for this django project. This should be done to manage the dependencies and packages used in the project. If you are not aware of the virtual environment and django setup, you can follow up with this [post](https://mr-destructive.github.io/techstructive-blog/django-setup-script/). #### Creating a runtime.txt file Now, Firstly we need to specify which type and version of language we are using. Since Django is a Python-based web framework, we need to select the python version in a text file. **runtime.txt** ``` python-3.9.5 ``` Here, the version can be anything as per your project and the packages installed. #### Creating requirements.txt file We'll first create a `requirements.txt` file for storing all the dependencies and packages installed in the application. This will help in installing dependencies while deploying the application. We can either use a `requirements.txt` file using `virtualenv` or a `Pipfile` using Pipenv. Both serve the same purpose but a bit differently. Assuming you are in an isolated virtual environment for this Django project, you can create a requirements.txt file using the below command: Make sure the virtualenv is activated before running the command: ``` pip freeze > requirements.txt ``` This will create a simple text file that contains the package names along with the versions used in the current virtual environment. A simple Django requirements file would look something like this: ``` asgiref==3.4.1 Django==3.2.11 pytz==2021.3 sqlparse==0.4.2 typing_extensions==4.0.1 ```  This is vanilla Django without any additional dependencies but if you have integrated other functionalities like Django Rest Framework, PostgreSQL, Crispy Forms, Schedulers, etc. there might be additional dependencies that become quite crucial for the smooth working of the project. If you are using pipenv, you don't need to make any efforts to manually activate and manage the virtual environment, the dependencies are installed and taken care of by the pipenv installer. You just need to make sure to install any package with `pipenv install` and not `pip install` for better and improved package tracking. So, that's all we need to take care of packages and keep a list of these integrated packages for the project. You need to update the requirements.txt file every time you install any new package or modify the dependencies for a project. Simply use the command `pip freeze > requirements.txt` in the activated virtual environment. #### Creating a Procfile Next up, we have the `Procfile`, a procfile is a special file that holds information about the processes to be run to start or activate the project. In our case, for django we need a web process that can manage the server. A Procfile is a simple file without any extension, make sure to write `Procfile` as it is the name of the file in the root folder of the project. Inside the file add the following contents: **Procfile** ```Procfile web: gunicorn <project_name>.wsgi ``` For the Procfile, Railway has a built [Procfile](https://github.com/railwayapp/starters/blob/master/examples/djangopy/Procfile) in the Django Template, you can refer to it and create it as follows: ```Procfile web: python manage.py migrate && gunicorn <project_name>.wsgi ``` As we can see we have defined the `web` process using `gunicorn`, [Gunicorn](https://pypi.org/project/gunicorn/) is a python package that helps in creating WSGI HTTP Server for the UNIX operating systems. So, we need to install the package and update the package dependency list. ``` pip install gunicorn pip freeze > requirements.txt ``` #### Configuring Environment Variables We need to keep our secrets for the django project out of the deployed code and in a safe place, we can create environment variables and keep them in a `.env` file which will be git-ignored. We do not want this `.env` file to be open source and thus should not be committed. We first need to create a new secret key if you already have a GitHub repository, chances are you would have committed the default secret key open for the world to see, it is an insecure way of deploying django apps in production. This should generate a secret key that only you know now. So, just copy the key without the quotes and create a file `.env` in the root project folder. **.env** ``` SECRET_KEY=76419fd6885a677f802fd1d2b5acd0188e23e001042b05a8 ``` The `.env` file should also be added to the `.gitignore` file, so simply append the following in the `.gitignore` file ``` .env ``` This is just one of the environment variables in our django project, further, we will also be adding a few other variables like database credentials, debug status, etc. We have now created environment variables for our django application, we now need a way to parse these environment variables into the Django project. #### Parsing Environment variables using python-dotenv We will use [python-dotenv](https://pypi.org/project/python-dotenv/) to parse variables into the django settings configurations like `SECRET_KEY` and `DATABASES`. - Install `python-dotenv` with pip with the command : ``` pip install python-dotenv ``` We need to then modify the default variables in the `settings.py` file. Firstly, we will load in the `.env` file for accessing the environment variables for the configuration of the project. Append the following code, to the top of the `settings.py` file, make sure don't override the configuration so remove unnecessary imports and configurations. ``` python # <project_name>/settings.py import os from dotenv import load_dotenv BASE_DIR = Path(__file__).resolve().parent.parent load_dotenv(os.path.join(BASE_DIR, ".env")) ``` We have imported the package `dotenv` basically the `python-dotenv` into the `settings.py` file and the module `os` for loading the `.env` file. The `load_dotenv` function helps in loading the `key-value` pairs which are the configuration variables that can act as actual environment variables. We provide in a file to the [load_dotenv](https://saurabh-kumar.com/python-dotenv/) function which is the `.env` file in our case, you can pick up any location for the `.env` file but make sure to change the location while parsing the file into the `load_dotenv` function. After loading the variables into the `settings.py` file, we now need to access those variables and set the appropriate variables the configuration from the variables received from the `load_dotenv` function. The `os.getenv` function to access the environment variables. The `os.getenv` function takes a parameter as the `key` for the environment variable and returns the value of the environment variable. ``` python SECRET_KEY = os.getenv("SECRET_KEY") ``` We have secretly configured the `SECRET_KEY` for the django project. If you have any other variables as simple key-value pairs like `AUTH` passwords, username, etc. you can use this method to get the configuration variables. #### Add gitignore file We would need a `.gitignore` file for only committing the project and pushing it to the remote repository. Here, we will set up a minimal `.gitignore` file in our root repository. Here's a sample `.gitignore` for a minimal django project. ```gitignore .env/ .venv/ env/ venv/ *.env *.pyc db.sqlite3 staticfiles/ ``` If you want a full-fledged `.gitignore` for a complex django project, you can take the reference from Jose Padilla's [gitignore Template](https://github.com/jpadilla/django-project-template/blob/master/.gitignore) for a django project. That would be good to go for creating and serving up the project while deploying the project on Railway App. ### Git Commit the Django Project Now, we can safely commit the code and push it to a GitHub repository. This will make sure you have a basic django proejct on GitHub from which we can build the Railway app into deployment. ``` git status git add . git commit -m "config for railway deployment" ``` Carefully check the status of the git repository before committing and make sure you don't forget anything by mistake, only commit the files which you think are important and ignore the rest. ``` git remote add rail https://github.com/Mr-Destructive/django-blog/tree/railway git push rail railway ```  ### Creating the Railway project Now, since we have a django project nicely configured and setup on GitHub, we can pull out a railway project and fetch the project from the GitHub repository. <video width="800" height="450" controls> <source src="https://res.cloudinary.com/techstructive-blog/video/upload/v1652970864/blog-media/django-deployment/railway_project_init.webm" type="video/mp4"> </video>  #### Spinning up the database We also need a database that we can attach in the django project in our Railway application. We can integrate a `PostgreSQL` database as a service running in our Railway project instance. We can do that by locating the `+New` tab on the project dashboard and attaching a `Database` in the drop-down menu. <video width="800" height="450" controls> <source src="https://res.cloudinary.com/techstructive-blog/video/upload/v1652963718/blog-media/django-deployment/postgres_spinup_railway_d2xkpt.mp4" type="video/mp4"> </video> After creating a database service, we need the credentials of the database or the `DATABASE_URL` of the PostgreSQL in order to integrate it into the django settings. We can locate into the `Connect` of the PostgreSQL service and grab the URL of the database. This can be stored in the main django application serves as an environment variable. ``` DATABASE_URL=postgresql://postgres:SE74bEw@containers-51.railway.app:6919/railway ```  In the Django Starter Project provided by Railway, we should have a PostgreSQL database integrated as a Service. So, we can consume the database in our django project. We will add the `DATABASE_URL` config variable into the `.env` file and also add it into the main Django project service so that it can communicate to the database. You need to copy the `DATABSE_URL` into our local setup file(`.env` file). Copy the Database URL and paste it into the `.env` file as follows: ```env DATABASE_URL=postgres://sjxgipufegmgsw:78cbb568e@ec2-52-4-104-184.compute-1.amazonaws.com:5432/dbmuget ``` The format for the Postgres URL is as follows: ``` postgresql://[user[:password]@][netloc][:port][/dbname] ``` #### Loading Database configuration Databases are a bit different as compared to other simple configurations in django. We need to make a few adjustments to the default database configuration. We will install another package `dj-database-url` to configure `DATABASE_URL`. Since the databse_url has a few components we need a way to extract the details like the `hostname`, `port`, `database_name`, and `password`. Using the `dj-database-url` package we have a few functions that can serve the purpose. ``` pip install dj-database-url ``` At the end of your `settings.py` file, append the following code. ``` python import dj_database_url DATABASE_URL = os.getenv("DATABASE_URL") DATABASES = { "default": dj_database_url.config(default=DATABASE_URL, conn_max_age=1800), } ``` We would need an adapter for making migrations to the `PostgreSQL` database i.e. the `psycopg2` package. This is a mandatory step if you are working with `postgres` database. This can be installed with the simple pip install: ``` pip install psycopg2 # If it does not work try pip install psycopg2-binary # if still error persists try installing setuptools pip install -U setuptools pip install psycopg2 ``` Now, that we have configured the database, we can now apply migrations to the new database of Postgres provided by Railway. We will simply run the migrate command and in the Railway Project the PostgreSQL database would have been modified and an appropriate schema should be applied. ``` python manage.py migrate ```  Make sure to update the `requirements.txt` file before pushing the project to Railway app for making sure everything works as expected. Since we have installed a few additional packages that are directly used in the `settings.py` file, we need to run the `pip freeze` command to update the `requiremnets.txt` file. #### Serving Static Files Now, if you have some static files like `CSS`, `Javascript`, or `images`, you need to configure the static files in order to serve them from the Railway app server. We will require another config variable for collecting the static files from the selected repository. ```python STATIC_URL = "static/" STATICFILES_DIRS = [os.path.join(BASE_DIR, "static")] STATIC_ROOT = os.path.join(BASE_DIR, "staticfiles") ``` Here, if you have served your static files from the `static` folder in the root directory of your django project, you can add the above code in the settings.py file. We will collect all static files in the folder along with the default static files provided by django in the `staticfiles` directory. Run the following command if you want to test whether the static files are properly collected and served. ``` python manage.py collectstatic ```  So, this command will collect all the static files and store them in a single place. We see that the files from the admin section are also copied as well as the custom static files from the project configuration. Now, we can move on to set the config variable for the Railway app in the Variables Tab. ``` DISABLE_COLLECTSTATIC = 0 ``` We can set the `DISABLE_COLLECTSTATIC` variable as `0` or `1` indicating whether to disable it or not. We have currently enabled the static file collection while deploying the app but you can set it to `1` to disable the collection of static files. Since I first tested the application on Railway, the static files don't work as expected, we need another package to make sure the staticfiles are served property. We will be installing the `whitenoise` package which serves as the middleware for serving the static files. ``` pip install whitenoise ``` Add the whitenoise middleware `whitenoise.middleware.WhiteNoiseMiddleware` to the `MIDDLEWARE` list in the `settings.py` file. ```python MIDDLEWARE = [ ... ... ... 'whitenoise.middleware.WhiteNoiseMiddleware', ] ``` That should be enough to make the most of the deployment on Railway app. You will have to make a few adjustments as per your requirements and project. ## Deploy to GitHub Finally, we will have all the pieces connected, only we need to push the code to the GitHub repository in order to trigger a build on the railway app. So, we make sure we commit every critical thing that are in our django project and include every sensitiv information in the gitignore file. ``` pip freeze > requirements.txt ``` This step is quite important because you need to make sure that all the packages are listed in the `requirements.txt` file else you will have to wait for the build to fail and redeploy. Make sure the server is running first on your local machine, remember the server will be set up from scratch but the database will already have applied migrations if you have applied migrations before after connecting the Railway database service. ``` python manage.py collectstatic python manage.py runserver ``` This will set up the origin of the remote repository that will be pushing the project code. Next, make sure to commit the code which will contain all the required stuff for deploying the code. Checklist for deploying the code ``` - requirements.txt - Procfile - runtime.txt - django-project - environment variables / config variables - static file configuration - database configuration - migrate schema of database - gitignore file for ignoring virtualenvs, .env file, staticfiles, etc ``` ``` git push origin main origin -> remote repository URL main -> branch name ```  ## Railway CLI Railway also provides a cool CLI, it has some quite handy features like creating and managing services, local development environment, etc. We'll just dive into a few nice features of the CLI tool. We'll first install the CLI on our local system, for that the guide is quite limited in a way for a few options to choose from like `npm`, `shell`, and `scoop`. For me, the shell was the way to go, but it had a few issues with permission, so I made a few changes in the [install.sh](https://github.com/railwayapp/cli/blob/master/install.sh) script ran on my machine and it worked fine.  Now, that we have the `Railway CLi` set up we can run a few commands like: ``` railway login OR # if it doesn't work railway login --browserless OR # with node npx railway login ```  This will sign you in with your railway account. We now need to link our project and execute and work around the command in that project from the railway app. To link a project from a railway account, you have to use the `link` command. The link command takes in a parameter as the project id which can be obtained from the project dashboard settings tab. ``` railway link <proejct_id> ``` <video width="800" height="450" controls> <source src="https://res.cloudinary.com/techstructive-blog/video/upload/v1652969451/blog-media/django-deployment/railway-proejct-link.webm" type="video/mp4"> </video> Now, we can explore some more commands like `run`, `up`, and so on. The `run` command is pretty solid and provides an interface to run commands for your project for instance in Django, create a superuser, manage dependencies, collectstatic files, etc. This allows us to set up and run commands into the actual production environment just from the CLI. ``` # railway run (command to be executed) railway run python manage.py createsuperuser # deploy the project railway up ``` <video width="800" height="450" controls> <source src="https://res.cloudinary.com/techstructive-blog/video/upload/v1652968693/blog-media/django-deployment/railway-cli-run-cmd.webm" type="video/mp4"> </video> So, this was all about creating and deploying a django application on Railway. Here are some reference links: - [GitHub Repository](https://github.com/Mr-Destructive/django-blog/tree/railway) - [Live Webpage](https://django-blog-production.up.railway.app/) ## Conclusion So, that's how we deploy a Django project on the Railway app. We have seen two of the many ways to create and deploy a Django application on Railway. We also integrated a PostgreSQL service provided by Railway in our Django Project. Using the Railway CLI, we were able to manage and create Projects from the terminal, also interaction of the railway project was achieved along with the production build from CLI. Hopefully, you liked the above tutorial, if you have any questions. feedback, or queries, you can contact me on the Social handles provided below. Thank you for reading and till the next post Happy Coding :) https://www.meetgor.com/posts/django-deploy-heroku Introduction Django projects are quite easy to build and simple to understand, you might have created a Django application and wanted to show it to the world? Y Sun, 15 May 2022 00:00:00 UTC ## Introduction Django projects are quite easy to build and simple to understand, you might have created a Django application and wanted to show it to the world? You can deploy a basic Django application with a database(PostgreSQL) with Heroku. It provides a decent free tier with some great features and add-ons. A free tier Heroku account has a limitation of 5 apps, limited data in the database, limited connections to the server per month, and so on. Though the free tier is not a great option for bigger applications, it suits really well for smaller scale and ide projects, so we will be looking into the details of how to deploy a Django application to [Heroku](https://heroku.com/) which is a Platform as Service (PaS). This series will be an extension of the series [Django basics](https://techstructiveblog.hashnode.dev/series/django-basics) which covered the basics of the Django framework, we covered from basic Django fundamentals to building a CRUD API. In this series, we will be exploring some platforms for giving a better understanding of how things work and understanding a few components that were left as default while understanding the basics of Django. Let's get started with [Django Deployment](https://techstructiveblog.hashnode.dev/series/django-deployment)! ## Creating a Django Application For deploying an app, we definitely need an app, we need to create a basic Django application to deploy on the web. We'll be creating a simple blog application with a couple of views and a simple model structure. As for the database, we'll be using Postgres as Heroku has an add-on for it and it is pretty easy to configure. ### Set up a virtual environment We need to set up a virtual environment in order to keep the Django project neat and tidy by managing the project-specific dependencies and packages. We can use the [virtualenv](https://pypi.org/project/virtualenv/) package to isolate a python project from the rest of the system. ``` # install the virtualenv package pip install virtualenv # create a virtual env for the project virtualenv .venv # activate the virtualenv Windows: .venv\Scripts�ctivate Linux/macOS: source .venv/bin/activate ``` This will set up the project nicely for a Django project, you now install the core Django package and get started with creating a Django application. ```bash # install django pip install django # start a django project django-admin startproject blog . cd blog # create a application in django project python manage.py createapp api # Create some models, views, URLs, templates # run the server python manag.py runserver ``` We assume you already have a Django project configured with some basic URLs, views and templates or static files as per your project and requirements, for this tutorial I will be using the simple blog application from my previous Django tutorials as a reference. You can follow along with my [Django Basics](https://techstructiveblog.hashnode.dev/series/django-basics) series and refer to the Blog Application project on [GitHub](https://github.com/Mr-Destructive/django-blog). ## Configuring the Django Application Make sure to create and activate the virtual environment for this django project. This should be done to manage the dependencies and packages used in the project. If you are not aware of the virtual environment and django setup, you can follow up with this [post](https://mr-destructive.github.io/techstructive-blog/django-setup-script/). ### Creating a runtime.txt file Now, Firstly we need to specify which type and version of language we are using. Since Django is a Python-based web framework, we need to select the python version in a text file. **runtime.txt** ``` python-3.9.5 ``` Here, the version can be anything as per your project and the packages installed. ### Creating requirements.txt file We'll first create a `requirements.txt` file for storing all the dependencies and packages installed in the application. This will help in installing dependencies while deploying the application. We can either use a `requirements.txt` file using `virtualenv` or a `Pipfile` using Pipenv. Both serve the same purpose but a bit differently. Assuming you are in an isolated virtual environment for this Django project, you can create a requirements.txt file using the below command: Make sure the virtualenv is activated before running the command: ``` pip freeze > requirements.txt ``` This will create a simple text file that contains the package names along with the versions used in the current virtual environment. A simple Django requirements file would look something like this: ``` asgiref==3.4.1 Django==3.2.11 pytz==2021.3 sqlparse==0.4.2 typing_extensions==4.0.1 ```  This is vanilla Django without any additional dependencies but if you have integrated other functionalities like Django Rest Framework, PostgreSQL, Crispy Forms, Schedulers, etc. there might be additional dependencies that become quite crucial for the smooth working of the project. If you are using pipenv, you don't need to make any efforts to manually activate and manage virtual environment, the dependencies are installed and taken care of by the pipenv installer. You just need to make sure to install any package with `pipenv install` and not `pip install` for better and improved package tracking. So, that's all we need to take care of packages and keep a list of these integrated packages for the project. You need to update the requirements.txt file every time you install any new package or modify the dependencies for a project. Simply use the command `pip freeze > requirements.txt` in the activated virtual environment. ### Creating a Procfile Next up, we have the `Procfile`, a procfile is a special file that holds information about the processes to be run to start or activate the project. In our case, for django we need a web process that can manage the server. A Procfile is a simple file without any extension, make sure to write `Procfile` as it is as the name of the file in the root folder of the project. Inside the file add the following contents: **Procfile** ```Procfile web: gunicorn <project_name>.wsgi ``` As we can see we have defined the `web` process using `gunicorn`, [Gunicorn](https://pypi.org/project/gunicorn/) is a python package that helps in creating WSGI HTTP Server for the UNIX operating systems. So, we need to install the package and update the package dependency list. ``` pip install gunicorn pip freeze > requirements.txt ``` That would be good to go for creating and serving up the project while deploying the project on Heroku. ## Creating a Heroku App A Heroku App is basically like your Django Project, you can create apps for deploying your django projects on Heroku. You are limited to 5 apps on the Free tier but that can be expanded on the paid plans.  The name of your heroku app should be unique globally, you need to try a few combinations before a good one fits your project. This name has no significance on your django project code, though it would be the name from which you would access the web application as a name `<app-name>.herokuapp.com`. So, choose it wisely if you are not attaching a custom domain. You can attach a custom domain, you can navigate to the `domain` section in the settings tab. ## Setting up the database To set up and configure a database in django on Heroku, we need to manually acquire and attach a PostgreSQL add-on to the heroku app. - Firstly locate to the Resources Tab in your Heroku app. - Search `postgres` in the Add-ons Search bar - Click on the `Heroku Postgres` Add-on - Submit the Order Form and you have the add-on enabled in the app.   Alternately, you can use the Heroku CLI, ### Heroku CLI Setup You can use the Heroku CLI which is a command-line interface for managing and creating Heroku applications. You need to first install the CLI by heading towards the [heroku documentation](https://devcenter.heroku.com/articles/heroku-command-line). Once the CLI is installed, you need to login into your Heroku account by entering the following command: ``` heroku login ```  This will allow us to work with heroku commands and manage our heroku application from the command line itself. The below command will create a add-on for `heroku-postgres` for the application provided as the parameter options ``` heroku addons:create heroku-postgresql:hobby-dev --app <app_name> ```  This should hopefully add a fresh instance of a postgres database for your heroku app. You can now configure the database for your project, we simply need the Database URL from the heroku app dashboard. We'll see how to configure the environment variables in Django for Heroku to keep your secrets like the `SECRET_KEY`, `DATABSE_URL`, etc. If you want MySQL as a database, you can check out the [ClearDB](https://devcenter.heroku.com/articles/cleardb) Add-On for Heroku to simply attach it like Postgres on your Heroku application. Also, if you wish to add [MongoDB](https://www.mongodb.com/compatibility/mongodb-and-django) into your Django application on Heroku, you can use [Object Rocket](https://devcenter.heroku.com/articles/ormongo)(OR Mongo). It is not available in the free tier though, unlike PostgreSQL and MySQL. ## Configuring Environment Variables We need to keep our secrets for the django project out of the deployed code and in a safe place, we can create environment variables and keep them in a `.env` file which will be git-ignored. We do not want this `.env` file to be open source and thus should not be committed. We first need to create a new secret key if you already have a GitHub repository, chances are you would have committed the default secret key open for the world to see, it is an insecure way of deploying django apps in production. So, open up a terminal and launch a python REPL: ``` python import secrets secrets.token_hex(24) ```  This should generate a secret key that only you know now. So, just copy the key without the quotes and create a file `.env` in the root project folder. **.env** ``` SECRET_KEY=76419fd6885a677f802fd1d2b5acd0188e23e001042b05a8 ``` The `.env` file should also be added to the `.gitignore` file, so simply append the following in the `.gitignore` file ``` .env ``` This file is only created to test the project locally, so we need to also make this key available on heroku. For doing that we need to add Config Variables to the heroku app. - Locate to the Settings Tab in your Heroku Application Dashboard - We have the `Config Vars` section in the located tab = We need to reveal those variables and we will be able to see all the variables.  We already have a `DATABASE_URL` variable declared when we attached the `django-postgres` database to our application. We will now add one more variable to the Application, i.e. the `SECRET_KEY`. Simply, enter the name of the key and also enter the value of the key, so basically a key-value pair. Simply click on the `Add` button and this will add the variable to your application.  You also need to copy the `DATABSE_URL` into our local setup file(`.env` file). Copy the Database URL and paste it into the `.env` file as follows: ```env DATABASE_URL=postgres://sjxgipufegmgsw:78cbb568e@ec2-52-4-104-184.compute-1.amazonaws.com:5432/dbmuget ``` The format for the postgres URL is as follows: ``` postgresql://[user[:password]@][netloc][:port][/dbname] ``` We have now created environment variables for our django application and also added config vars in the heroku app, we now need a way to parse these environment variables into the Django project. ### Parsing Environment variables using python-dotenv We will use [python-dotenv](https://pypi.org/project/python-dotenv/) to parse variables into the django settings configurations like `SECRET_KEY` and `DATABASES`. - Install `python-dotenv` with pip with the command : ``` pip install python-dotenv ``` We need to then modify the default variables in the `settings.py` file. Firstly, we will load in the `.env` file for accessing the environment variables for the configuration of the project. Append the following code, to the top of the `settings.py` file, make sure don't override the configuration so remove unnecessary imports and configurations. ``` python # <project_name>/settings.py import os from dotenv import load_dotenv BASE_DIR = Path(__file__).resolve().parent.parent load_dotenv(os.path.join(BASE_DIR, ".env")) ``` We have imported the package `dotenv` basically the `python-dotenv` into the `settings.py` file and the module `os` for loading the `.env` file. The `load_dotenv` function helps in loading the `key-value` pairs which are the configuration variables that can act as actual environment variables. We provide in a file to the [load_dotenv](https://saurabh-kumar.com/python-dotenv/) function which is the `.env` file in our case, you can pick up any location for the `.env` file but make sure to change the location while parsing the file into the `load_dotenv` function. After loading the variables into the `settings.py` file, we now need to access those variables and set the appropriate variables the configuration from the variables received from the `load_dotenv` function. The `os.getenv` function to access the environment variables. The `os.getenv` function takes a parameter as the `key` for the environment variable and returns the value of the environment variable. ``` python SECRET_KEY = os.getenv("SECRET_KEY") ``` We have secretly configured the `SECRET_KEY` for the django project. If you have any other variables as simple key-value pairs like `AUTH` passwords, username, etc. you can use this method to get the configuration variables. ### Loading Database configuration Databases are a bit different as compared to other simple configurations in django. We need to make a few adjustments to the default database configuration. We will install another package `dj-database-url` to configure `DATABASE_URL`. Since the databse_url has a few components we need a way to extract the details like the `hostname`, `port`, `database_name`, and `password`. Using the `dj-database-url` package we have a few functions that can serve the purpose. ``` pip install dj-database-url ``` At the end of your `settings.py` file, append the following code. ``` python import dj_database_url DATABASE_URL = os.getenv("DATABASE_URL") DATABASES = { "default": dj_database_url.config(default=DATABASE_URL, conn_max_age=1800), } ``` We would need an adapter for making migrations to the `PostgreSQL` database i.e. the `psycopg2` package. This is a mandatory step if you are working with `postgres` database. This can be installed with the simple pip install: ``` pip install psycopg2 # If it does not work try pip install psycopg2-binary # if still error persists try installing setuptools pip install -U setuptools pip install psycopg2 ``` Now, that we have configured the database, we can now apply migrations to the fresh database of postgres provided by heroku. We will simply run the migrate command and in the heroku app the PostgreSQL database would have been modified and an appropriate schema should be applied. ``` python manage.py migrate ```  Make sure to update the `requirements.txt` file before pushing the project to Heroku for making sure everything works as expected. Since we have installed a few additional packages that are directly used in the `settings.py` file, we need to run the `pip freeze` command to update the `requiremnets.txt` file. ## Serving Static Files Now, if you have some static files like `CSS`, `Javascript`, or `images`, you need to configure the staticfiles in order to serve them from the heroku server. We will require another config variable for collecting the static files from the selected repository. ```python STATIC_URL = "static/" STATICFILES_DIRS = [os.path.join(BASE_DIR, "static")] STATIC_ROOT = os.path.join(BASE_DIR, "staticfiles") ``` Here, if you have served your static files from the `static` folder in the root directory of your django project, you can add the above code in the settings.py file. We will collect all static files in the folder along with the default static files provided by django in the `staticfiles` directory. Run the following command if you want to test whether the static files are properly collected and served. ``` python manage.py collectstatic ```  So, this command will collect all the static files and store them in a single place. We see that the files from the admin section are also copied as well as the custom static files from the project configuration. Now, we can move on to set the config variable for the heroku app. ``` DISABLE_COLLECTSTATIC = 0 ``` We can set the `DISABLE_COLLECTSTATIC` variable as `0` or `1` indicating whether to disable it or not. We have currently enabled the static file collection while deploying the app but you can set it to `1` to disable the collection of static files.  Since I first tested the application on heroku, the static files don't work as expected, we need another package to make sure the staticfiles are served property. We will be installing the `whitenoise` package which serves as the middleware for serving the static files. ``` pip install whitenoise ``` Add the whitenoise middleware `whitenoise.middleware.WhiteNoiseMiddleware` to the `MIDDLEWARE` list in the `settings.py` file. ````python MIDDLEWARE = [ ... ... ... 'whitenoise.middleware.WhiteNoiseMiddleware', ] ``` That should be enough to make the most of the deployment on heroku. You will have to make a few adjustments as per your requirements and project. ## Deploy from GitHub We are now all set to deploy the application on Heroku, we can use the `Connect to GitHub` or `Heroku CLI` to push the code into production. Heroku CLI is quite easy with a few sets of commands but if your project is deployed on GitHub, you can straightaway let the deployment start the build on a push to a specific branch. This becomes quite automotive and easy to scale while deploying a large-scale application. ``` pip freeze > requirements.txt ``` This step is quite important because you need to make sure that all the packages are listed in the `requirements.txt` file else you will have to wait for the build to fail and redeploy. Make sure the server is running first on your local machine, remember the server will be set up from scratch but the database will already have applied migrations if you have applied migrations before after connecting the Heroku Postgres database. ``` python manage.py collectstatic python manage.py runserver ``` This will set up the origin of the remote repository that will be pushing the project code. Next, make sure to commit the code which will contain all the required stuff for deploying the code. Checklist for deploying the code ``` - requirements.txt - Procfile - runtime.txt - django-project - environment variables / config variables - static file configuration - database configuration - migrate schema of database - gitignore file for ignoring virtualenvs, .env file, staticfiles, etc ``` here's a sample `.gitignore` for a minimal django project. ```gitignore .env/ .venv/ env/ venv/ *.env *.pyc db.sqlite3 staticfiles/ ``` If you want a full-fledged `.gitignore` for a complex django project, you can take the reference from Jose Padilla's [gitignore Template](https://github.com/jpadilla/django-project-template/blob/master/.gitignore) for a django project. ### Git Commit the Django Project ``` git status git add . git commit -m "config for heroku deployment" ``` Carefully check the status of the git repository before committing and make sure you don't forget anything by mistake, it won't a big problem but it would mess up the build process.  After committing the code, we can now push the code to GitHub. We first need to set the remote repository reference to be able to push the code to it. ``` git remote add origin https://github.com/<username>/<repo_name> ``` This will set up the `origin` as the remote repository on GitHub. Once the remote repository is created, we can move into the push of the git repository that will trigger the build. First, navigate to the `Deploy` section in the heroku app's dashboard where we want to connect the `GitHub` repository and allow the automatic deploy from a branch in this case we have chosen the `main` branch. Due to some `Heroku` Internal Server Issues, the GitHub integration seems to have broken and isn't working as of May 2022, but it might work later. Below is a screenshot of my previous project deployed to Heroku using a GitHub repository.  ``` git push origin main ``` This will prompt you for your GitHub credentials and will deploy the commits to the remote repository on GitHub. This push on the main branch should also trigger the build process of the heroku app for this django project. You can navigate to the Activity section for the Build logs. If you have followed the article well, and your repository has all the correct configurations, the build will succeed, else chances are you might have missed a few things and the app might have crashed. You can debug your application build with the simple heroku CLI command: ``` heroku logs --tail -a <app_name> ``` This can be quite handy and saves a lot of time in understanding what went wrong in the build. It might be related to database migration, static files, python package not found, and some silly mistakes and errors that can be fixed after committing the code and pushing it to GitHub again. If you do not want a GitHub repository, you can directly push the code from the local git repository to the remote heroku app center. This will require us the Heroku CLI. ## Heroku CLI We can use the heroku CLI for pushing the code via the simple git repository. We can push the references via the branch and a remote repository on heroku to build our app. For this, we assume you have heroku installed and logged in. We will require the django project code and heroku cli to build the django web application. ```bash heroku git:remote -a <heroku_app_name> # for my case heroku git:remote -a blog-django-dep ```  After this, you can commit your code and the project as git repository. We have added the remote repository location on heroku, we can now simply push the code to the remote repository. ``` git push heroku main ```  Here, `heroku` is the remote repository location and `main` is the branch of the repository. This will push the code to the repository and thereafter create a build to deploy the Django project as a Heroku application.  You can hit the `Open App` button on the top right corner and there should be your deployed Django application.  ## A Few Tricks and Gotchas There are a few tricks and issues that you might pop into while deploying a django project on heroku, especially the build process. It requires a few iterations to get the complete app setup. ### Run command from the Dashboard console If you don't have heroku CLI set up and want to fix a few things on the pushed code to the heroku app, you can use the `Run Console` option from the `More` Tab on the top right corner of theApplication dashboard. This is a great way to run migrations, configure static files or tweak a few things here and there without messing up the code on GitHub or the remote git repositories.    ### Deploy with Docker You can even use the docker container to deploy a Django application on Heroku. It is a great way of learning a lot of deployment strategies and techniques using Docker. You'll get familiar with interesting concepts like virtualization, and containerization, and also learn Docker on the way. You can follow this tutorial on [Deploying Django applications with Docker on Heroku](https://testdriven.io/blog/deploying-django-to-heroku-with-docker/). Also, you can check out the official Heroku documentation for [deploying python applications](https://devcenter.heroku.com/articles/deploying-python). ### What are Dynos? Dynos are simply web processes or workers that serve your web application. Dynos in Heroku are allocated based on the build process, once the slug is created a dyno is created as it runs on a VM container. This simply means there are limitations on how to use the web application and its sleep process. The hobby tier is sufficient for normal testing projects and side projects though you will have to pay and move into advance tiers to increase the dyno allocations and scaling of those web processes. It's not a simple thing to understand but to keep it simple, it might be a container to process the client's request and serve the page for a finite duration of the interaction. Also, your application will sleep after half an hour, if you try to reload the application every half an hour it will consume your resource allocation for the month and this is how the tiers and divided for paid services on Heroku. You can check out the detail over [here](https://www.heroku.com/pricing#containers). ## Conclusion So, that is one of the ways we can deploy a Django application on Heroku with the PostgreSQL database. You can find the [django-blog project] on [GitHub] for following along with the deployment process. In the next few parts of the series, we will be hopefully covering other platforms where you can deploy a Django o application. Hopefully, you liked the above tutorial, if you have any questions. feedback, or queries, you can contact me on the Social handles provided below. Thank you for reading and till the next post Happy Coding :) https://www.meetgor.com/posts/golang-operators Introduction In this 13th part of the series, we will be exploring the fundamentals of operators in Golang. We will be exploring the basics of operators and the Sat, 07 May 2022 00:00:00 UTC ## Introduction In this 13th part of the series, we will be exploring the fundamentals of operators in Golang. We will be exploring the basics of operators and the various types like Arithmetic, Bitwise, Comparison, Assignment operators in Golang. Operators are quite fundamentals in any programming language. Operators are basically expressions or a set of character(s) to perform certain fundamental tasks. They allow us to perform certain trivial operations with a simple expression or character. There are quite a few operators in Golang to perform various operations. ## Types of Operators Golang has a few types of operators, each type providing particular aspect of forming expressions and evaluate conditions. 1. Bitwise Operators 2. Logical Operators 3. Arithmetic Operators 4. Assignment Operators 5. Comparison Operators ### Bitwise Operators Bitwise Operators are used in performing operations on binary numbers. We can perform operation on a bit level and hence they are known as bitwise operators. Some fundamental bitwise operators include, `AND`, `OR`, `NOT`, and `EXOR`. Using this operators, the bits in the operands can be manipulated and certain logical operations can be performed. ```go package main import "fmt" func main() { x := 3 y := 5 // 3 -> 011 // 5 -> 101 fmt.Println("X AND Y = ", x & y) fmt.Println("X OR Y = ", x | y) fmt.Println("X EXOR Y = ", x ^ y) fmt.Println("X Right Shift 1 = ", x >> 1) fmt.Println("X Right Shift 2 = ", x >> 2) fmt.Println("Y Left Shift 1 = ", y << 1) } ``` ``` $ go run bitwise/main.go X AND Y = 1 X OR Y = 7 X EXOR Y = 6 X Right Shift 1 = 1 X Right Shift 2 = 0 Y Left Shift 1 = 10 ``` We use the `&` (AND operator) for performing AND operations on two operands. Here we are logically ANDing `3` and `5` i.e. `011` with `101` so it becomes `001` in binary or 1 in decimal. Also, the `|` (OR operator) for performing logical OR operation on two operands. Here we are logically ORing `3` and `5` i.e. `011` with `101` so it becomes `111` in binary or 7 in decimal. Also the `^` (EXOR operator) for performing logical EXOR operation on two operands. Here we are logically EXORing `3` and `5` i.e. `011` with `101` so it becomes `110` in binary or 6 in decimal. We have a couple of more bitwise operators that allow us to shift bits in the binary representation of the number. We have two types of these shift operators, right sift and left shift operators. The main function of these operator is to shift a bit in either right or left direction. In the above example, we have shifted `3` i.e. `011` to right by one bit so it becomes `001`. If we would have given `x >> 2` it would have become `0` since the last bit was shifted to right and hence all bits were 0. Similarly, the left shift operator sifts the bits in the binary representation of the number to the left. So, in the example above, `5` i.e. `101` is shifted left by one bit so it becomes `1010` in binary i.e. 10 in decimal. This was a basic overview of bitwise operators in Golang. We can use these basic operators to perform low level operations on numbers. ### Comparison Operators This type of operators are quite important and widely used as they form the fundamentals of comparison of variables and forming boolean expressions. The comparison operator is used to compare two values or expressions. ```go package main import "fmt" func main() { a := 45 b := 12 fmt.Println("Is A equal to B ? ", a == b) fmt.Println("Is A not equal to B ? ", a != b) fmt.Println("Is A greater than B ? ", a > b) fmt.Println("Is A less than B ? ", a < b) fmt.Println("Is A greater than or equal to B ? ", a >= b) fmt.Println("Is A less than or equal to B ? ", a <= b) ``` ``` $ go run comparison/main.go Is A equal to B ? false Is A not equal to B ? true Is A greater than B ? true Is A less than B ? false Is A greater than or equal to B ? true Is A less than or equal to B ? false ``` We use simple comparison operators like `==` or `!=` for comparing if two values are equal or not. The expression `a == b` will evaluate to `true` if the values of both variables or operands are equal. However, the expression `a != b` will evaluate to `true` if the values of both variables or operands are not equal. Similarly, we have the `<` and `>` operators which allow us to evaluate expression by comparing if the values are less than or grater than the other operand. So, the expression `a > b` will evaluate to `true` if the value of `a` is greater than the value of `b`. Also the expression `a < b` will evaluate to `true` if the value of `a` is less than the value of `b`. Finally, the operators `<=` and `>=` allow us to evaluate expression by comparing if the values are less than or equal to and greater than or equal to the other operand. So, the expression `a >= b` will evaluate to `true` if the value of `a` is greater than or if it is equal to the value of `b`, else it would evaluate to `false`. Similarly, the expression `a <= b` will evaluate to `true` if the value of `a` is less than or if it is equal to the value of `b`, else it would evaluate to `false`. These was a basic overview of comparison operators in golang. ### Logical Operators Next, we move on to the logical operators in Golang which allow to perform logical operations like `AND`, `OR`, and `NOT` with conditional statements or storing boolean expressions. ```go package main import "fmt" func main() { a := 45 b := "Something" fmt.Println(a > 40 && b == "Something") fmt.Println(a < 40 && b == "Something") fmt.Println(a < 40 || b == "Something") fmt.Println(a < 40 || b != "Something") fmt.Println(!(a < 40 || b != "Something")) } ``` ``` $ go run logical/main.go true false true false true ``` Here, we have used logical operators like `&&` for Logical AND, `||` for logical OR, and `!` for complementing the evaluated result. The `&&` operation only evaluates to `true` if both the expressions are `true` and `||` OR operator evaluates to `true` if either or both the expressions are `true`. The `!` operator is used to complement the evaluated expression from the preceding parenthesis. ### Arithmetic Operators Arithmetic operators are used for performing Arithmetic operations. We have few basic arithmetic operators like `+`, `-`, `*`, `/`, and `%` for adding, subtracting, multiplication, division, and modulus operation in golang. ```go package main import "fmt" func main() { a := 30 b := 50 fmt.Println("A + B = ", a+b) fmt.Println("A - B = ", a-b) fmt.Println("A * B = ", a*b) fmt.Println("A / B = ", a/b) fmt.Println("A % B = ", a%b) } ``` ``` $ go run arithmetic/main.go A + B = 80 A - B = -20 A * B = 1500 A / B = 0 A % B = 30 ``` These are the basic mathematical operators in any programming language. We can use `+` to add two values, `-` to subtract two values, `*` to multiply to values, `/` for division of two values and finally `%` to get the remainder of a division of two values i.e. if we divide 30 by 50, the remainder is 30 and the quotient is 0. We also have a few other operators like `++` and `--` that help in incrementing and decrementing values by a unit value. Let's say we have a variable `k` which is set to `4` and we want to increment it by one, so we can definitely use `k = k + 1` but it looks kind of too long, we have a short notation for the same `k++` to do the same. ```go package main import "fmt" func main() { k := 3 j := 20 fmt.Println("k = ", k) fmt.Println("j = ", j) k++ j-- fmt.Println("k = ", k) fmt.Println("j = ", j) } ``` ``` $ go run arithmetic/main.go k = 3 j = 20 k = 4 j = 19 ``` So, we can see that the variable `k` is incremented by one and variable `j` is decremented by `1` using the `++` and `--` operator. ### Assignment Operators These types of operators are quite handy and can condense down large operations into simple expressions. These types of operators allow us to perform operation on the same operand. Let's say we have the variable `k` set to `20` initially, we want to add `30` to the variable `k`, we can do that by using `k = k + 30` but a more sophisticated way would be to use `k += 30` which adds `30` or any value provided the same variable assigned and operated on. ```go package main import "fmt" func main() { var a int = 100 b := 20 fmt.Println("a = ", a) fmt.Println("b = ", b) a += 30 fmt.Println("a = ", a) b -= 5 fmt.Println("b = ", b) a *= b fmt.Println("a = ", a) fmt.Println("b = ", b) a /= b fmt.Println("a = ", a) fmt.Println("b = ", b) a %= b fmt.Println("a = ", a) fmt.Println("b = ", b) } ``` ``` $ go run assignment/main.go a = 100 b = 20 a = 130 b = 15 a = 1950 b = 15 a = 130 b = 15 a = 10 b = 15 ``` From the above example, we are able to perform operations by using shorthand notations like `+=` to add the value to the same operand. These also saves a bit of time and memory not much but considerable enough. This allow us to directly access and modify the contents of the provided operand in the register rather than assigning different registers and performing the operations. That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/mr-destructive/100-days-of-golang/) GitHub repository. ## Conclusion So, from the following part of the series, we were able to learn the basics of operators in golang. Using some simple and easy to understand examples, we were able to explore different types of operators like arithmetic, logical, assignment and bitwise operators in golang. These are quite fundamental in programming in general, this lays a good foundation for working with larger and complex projects that deal with any kind of logic in it, without a doubt almost all of the applications do have a bit of logic attached to it. So, we need to know the basics of operators in golang. https://www.meetgor.com/posts/golang-math Introduction Moving on in the 100 days of golang series, we can take a look into the math package in golang's standard library. In programming, math is quite cr Mon, 25 Apr 2022 00:00:00 UTC ## Introduction Moving on in the 100 days of golang series, we can take a look into the math package in golang's standard library. In programming, math is quite critical aspect, we need to perform certain mathematical operations quite regularly so golang's standard library has a package for serving some quite commonly used math functions and procedures. We'll take a look at some of the basic and common functions which are available in the math package. ## Mathematical Constants We have some constants like `pi`, `e`, `Phi` already defined as constants in the math package of the standard library in golang. They have a precision till 15 digits stored in float64 values. ```go package main import ( "fmt" "math" ) func main() { fmt.Println("Pi = ", math.Pi) fmt.Println("E = ", math.E) fmt.Println("Phi = ", math.Phi) fmt.Println("Sqrt of 2 = ", math.Sqrt2) fmt.Println("Naturla Log 2 = ", math.Ln2) fmt.Println("Naturla Log 10 = ", math.Ln10) } ``` ``` $ go run basic-functions/constants.go Pi = 3.141592653589793 E = 2.718281828459045 Phi = 1.618033988749895 Sqrt of 2 = 1.4142135623730951 Naturla Log 2 = 0.6931471805599453 Naturla Log 10 = 2.302585092994046 ``` We can use these constants in trigonometric calculations and also in scientific computing. Further, you can get a list of all constants defined in the math package of the go standard library from the [documentation](https://pkg.go.dev/math#pkg-constants). ## Basic Math functions We have some quite basic and fundamental functions in the math package that can be used commonly in many programs. Let's take a look at a few of them. ### - Abs :parameters (float64) , returns float64 As the name suggest, the [Abs](https://pkg.go.dev/math#Abs) it returns the absolute result of a numbers. It takes a parameter as a float64 value and returns the absolute value of the provided number as a `float64` number. ```go package main import ( "fmt" "math" ) func main() { a := 45 b := 100 diff := a - b fmt.Println(diff) absolute_diff := math.Abs(float64(a) - float64(b)) fmt.Println(absolute_diff) } ``` ``` $ go run basic-functions/main.go -55 55 ``` As, we can see the `Abs` function takes in a float64 value and returns a absolute value of the given number that too a `float64` value. We need to cast the numbers `a` and `b` into `float64` as we have not provided the initial values and so the compiler has assigned the type to them as `int`. #### Type Casting We can caste a type into other by using the variable around the type name as `type_name(variable)`. In the above example we have converted the `int` value `45` into a `float64` as `float64(45)` which again yields `45` but as a float64 type. ```go foo := 77 fmt.Printf("Type of foo = %T ", foo) fmt.Println("foo = ", int(foo)) fmt.Println("String Cast: ", string(foo)) fmt.Println("Float Cast: ", float64(foo)) ``` Though not every type cannot be casted into due to quite oblivious reasons, for instance `77` or any other integer value (except for 0 or 1) cannot be converted into boolean value. **Hello Gopher! Just a small note, the math package almost deals with float64 types rather than int to avoid backwards compatibility to perform operations on floating point values which can be casted into integers rather than defining separate functions for decimal values and integers.** ### - Min/Max: parameters(float64) , returns float64 We can get the [minimum](https://pkg.go.dev/math#Min) and [maximum](https://pkg.go.dev/math#Max) value of the two numbers provided to the function. ```go var float64 a = 120 var float64 b = 54 minimum := math.Min(float64(a), float64(b)) maximum := math.Max(float64(a), float64(b)) fmt.Printf("Min of %v and %v is %v ", a, b, minimum) fmt.Printf("Max of %v and %v is %v ", a, b, maximum) ``` ``` $ go run basic-functions/main.go Min of 120 and 54 is 54 Max of 120 and 54 is 120 ``` ### - Pow : parameters(float64, float64) , returns float64 ### - Pow10: parameters(int) , returns float64 The [Pow](https://pkg.go.dev/math#Pow) function is used to get the exponential result of the base number. So, if we provide the values x and y, we would get the result as the number x raised to y. ```go var x float64 = 3 var y float64 = 4 z := math.Pow(x, y) z10 := math.Pow10(int(x)) fmt.Println("X ^ Y = ", z) fmt.Println("10 ^ X = ", z10) ``` ``` $ go run basic-functions/main.go X ^ Y = 81 10 ^ X = 1000 ``` We also have the [Pow10](https://pkg.go.dev/math#Pow10) function which works just like the pow function except the x value is 10 and we don't have to provide it, there is just one parameter as a integer which returns a float64 value. ### - Sqrt: parameters(float64) , returns float64 The [Sqrt](https://pkg.go.dev/math#Sqrt) function as the name suggest, it is used to get the square root value of a floating point value which returns a float64 value. ```go var k float64 = 125 sqrt_of_k := math.Sqrt(k) cbrt_of_k := math.Cbrt(k) fmt.Printf("Square root of %v = %v ", k, sqrt_of_k) fmt.Printf("Cube root of %v = %v ", k, cbrt_of_k) ``` ``` $ go run basic-functions/main.go Square root of 125 = 11.180339887498949 Cube root of 125 = 5 ``` ### - Trunc: parameters(float64) , returns float64 The [Truncate](https://pkg.go.dev/math#Trunc) function provides the way to round off a decimal value(float64) to an integer but it returns a value in `float64`. ```go var p float64 = 445.235 trunc_p := math.Trunc(p) fmt.Printf("Truncated value of %v = %v ", p, trunc_p) p = 123.678 trunc_p = math.Trunc(p) fmt.Printf("Truncated value of %v = %v ", p, trunc_p) ``` ``` $ go run basic-functions/main.go Truncated value of 445.235 = 445 Truncated value of 123.678 = 123 ``` ### - Ceil : parameters(float64) , returns float64 We also can use the [Ceil](https://pkg.go.dev/math#Ceil) function to roud up the value to the next integer value but the value is returned as `float64`. ```go var c float64 = 33.25 ceil_c := math.Ceil(c) fmt.Printf("Ceiled value of %v = %v ", c, ceil_c) c = 134.78 ceil_c = math.Ceil(c) fmt.Printf("Ceiled value of %v = %v ", c, ceil_c) ``` ``` $ go run basic-functions/main.go Ceiled value of 33.25 = 34 Ceiled value of 134.78 = 135 ``` ### - Trigonometric Functions Trigonometric functions are quite helpful that can help in intense mathematical computations in backend projects or precision dependent projects. We have functions [Sin](https://pkg.go.dev/math#Sin), [Cos](https://pkg.go.dev/math#Cos), [SinCos](https://pkg.go.dev/math#Sincos), [Tan](https://pkg.go.dev/math#Tan), hyperbolic functions in Trigonometric functions like [Sinh](https://pkg.go.dev/math#Sinh), [Cosh](https://pkg.go.dev/math#Cosh), [Tanh](https://pkg.go.dev/math#Tanh), and Inverse Trigonometric functions like [Asin](https://pkg.go.dev/math#Asin), [Asinh](https://pkg.go.dev/math#Asinh), etc. - Sin: parameters(float64) , returns float64 ```go package main import ( "fmt" "math" ) func main() { // basic trigonometric functions var x float64 = math.Pi / 2 sinx := math.Sin(x) cosx := math.Cos(x) tanx := math.Tan(x) fmt.Printf("Sin(%v) = %v ", x, sinx) fmt.Printf("Cos(%v) = %v ", x, cosx) fmt.Printf("Tan(%v) = %v ", x, tanx) // hyperbolic trigonometric functions var h float64 = math.Pi / 2 sinh := math.Sinh(h) cosh := math.Cosh(h) tanh := math.Tanh(h) fmt.Printf("Sinh(%v) = %v ", h, sinh) fmt.Printf("Cosh(%v) = %v ", h, cosh) fmt.Printf("Tanh(%v) = %v ", h, tanh) // Inverse Trigonometric functions var y float64 = -1 arc_sin := math.Asin(y) // -pi/2 radians or 90 degrees arc_cos := math.Acos(y) // pi randians or 180 degrees arc_tan := math.Atan(y) fmt.Printf("Sin^-1(%v) = %v ", y, arc_sin) fmt.Printf("Cos^-1(%v) = %v ", y, arc_cos) fmt.Printf("Tan^-1(%v) = %v ", y, arc_tan) ``` ``` $ go run basic-functions/trignometric.go Sin(1.5707963267948966) = 1 Cos(1.5707963267948966) = 6.123233995736757e-17 Tan(1.5707963267948966) = 1.6331239353195392e+16 Sinh(1.5707963267948966) = 2.3012989023072947 Cosh(1.5707963267948966) = 2.5091784786580567 Tanh(1.5707963267948966) = 0.9171523356672744 Sin^-1(-1) = -1.5707963267948966 Cos^-1(-1) = 3.141592653589793 Tan^-1(-1) = -0.7853981633974483 ``` Here we can see that the functions are working fine and giving a decently precise value. This might be enough for simple and smaller projects, though for higher precision and accuracy areas, other computations and programming is required to compute the values. ### - Exponential and Logarithmic Functions We also have the exponential and logarithmic functions defined in the math package to leverage computations realted to formulae that deal with logarithmic or exponential calculations. - Exp : parameters(flaot64) , returns float64 - Exp2 : parameters(flaot64) , returns float64 ```go package main import ( "fmt" "math" ) func main() { // exponential function var x float64 = 2 y := math.Exp(x) fmt.Println("e^x = ", y) var n float64 = 3.5 y = math.Exp2(n) fmt.Println("2^n = ", y) // Logarithmic function y = math.Log(x) fmt.Println("natural log x = ", y) n = 128 y = math.Log2(n) fmt.Println("Log2 of 100 = ", y) } ``` ``` $ go run basic-functions/expo_log.go e^x = 7.38905609893065 2^n = 11.31370849898476 natural log x = 0.6931471805599453 Log2 of 100 = 7 ``` Here, we have exponential functions such as `e^x` and `2^n` which might be useful in some common programming calculations. Also the logarithmic functions like `log x` which is natural log of x(base e), and `log2 n` which is logn to the base 2. ## The Random package The `random` sub-package in golang provides some great tools for working with random numbers and generating them. It provides exhaustive list of functions and types that help in generating pseudo random numbers. - Int : parameters() , returns int - Intn : parameters( int ) , returns int ```go package main import ( "fmt" "math/rand" ) func main() { // random integer generation x := rand.Int() fmt.Println(x) // random number generation till range for i := 0; i < 5; i++ { y := rand.Intn(10) fmt.Println(y) } } ``` ``` $ go run basic-functions/rand.go 5577006791947779410 7 7 9 1 8 ``` In the above example, we have used the [Int](https://pkg.go.dev/math/rand@go1.18.1#Int) function in the random sub-package of the math package which generates a pseudo random integer of the range dependent on the system architecture generally `int32` or `int64`. We get a huge number which is pseudo random i.e. not truly random. If you try to execute the program a couple of time, you would notice the number remains the same and we are calling it random? Well we need to dive into random numbers and seeding for a different part of the series for sure. The [Intn](https://pkg.go.dev/math/rand@go1.18.1#Intn) function also generates a pseudo random number but this time, we define the range of the upper boundary to generate them. It is not inclusive of the number provided i.e. we have provide the value `10` so the number `10` is not included in the range. It's called half open interval. It starts from 0 so the range becomes mathematically `[0, n)` if n is the number provided to the Intn function. ## The Bits package We also have a bit sub-package in the math package of the go standard library. This sub package is used for working around with bit manipulation and operations at the binary level. This is quite helpful in competitive programming , also in understanding the basics of data structures and fundamentals in computer science. - Add : parameters(uint, uint, uint) , returns uint, uint - Len : parameters(uint) , returns int - OnesCount : parameters(uint) , returns int ```go package main import ( "fmt" "math/bits" ) func main() { s, c := bits.Add(0, 9, 1) fmt.Printf("Sum = %d Carry = %d ", s, c) // (45) in decimal = (1 0 1 1 0 1) in binary var n uint = 45 length := bits.Len(n) ones_in_45 := bits.OnesCount(n) fmt.Printf("Minimum bits required to represent 45 = %d ", length) fmt.Printf("Set Bits in 45 = %d ", ones_in_45) } ``` ``` $ go run basic-functions/bit.go Sum = 10 Carry = 0 Minimum bits required to represent 45 = 6 Set Bits in 45 = 4 ``` Here, in the above example, we have used the bits sub pacakge in the math package, the [Add](https://pkg.go.dev/math/bits@go1.18.1#Add) function allows us to provide the two numbers and a carry bit on which it returns two values the sum and the carry. The sum is defined as the summation of `x + y + carry` the two numbers and the carry bit. The carry bit needs to be either 0 or 1. Also the value provided the function i.e. `x and y` need to be unsigned `uint` iorder to work with bits. We also have the [Len](https://pkg.go.dev/math/bits@go1.18.1#Len) function which returns the maximum number of bits required to represent the provided unsigned integer. We have used 45 which is equivalent to `10110` and hence the function returns `6` as teh number of bits. The [OnesCount](https://pkg.go.dev/math/bits@go1.18.1#OnesCount) Function is also similar but it returns the number of set bits(the `1` bit) in the number provided to it. We'll see this sub package in a separate section of its own. Bits is really a great pacakge to work with bits and low level manipulation of numbers in Golang. ## The Complex package The complex subpackage is really specific to the operation to the complex numbers and its operations. Using complex numbers with basic operations and trigonometric functions are provided in the package. ```go package main import ( "fmt" "math/cmplx" ) func main() { x := complex(5, 8) y := complex(3, 4) mod_x := cmplx.Abs(x) mod_y := cmplx.Abs(y) conj_x := cmplx.Conj(x) phase_x := cmplx.Phase(x) mod, phase := cmplx.Polar(x) fmt.Println("x = ", x) fmt.Println("Modulus of x = ", mod_x) fmt.Println("Modulus of y = ", mod_y) fmt.Println("Conjugate of x = ", conj_x) fmt.Println("Phase of x = ", phase_x) fmt.Printf("Polar Form : %v, %v ", mod, phase) } ``` ``` $ go run basic-functions/complex.go x = (5+8i) Modulus of x = 9.433981132056603 Modulus of y = 5 Conjugate of x = (5-8i) Phase of x = 1.0121970114513341 Polar Form : 9.433981132056603, 1.0121970114513341 ``` We have used the complex function to create complex numbers. The `cmplx` subpackage in the math package provides many functions to play with trignometric and simple operations with complex numbers. The [Abs](https://pkg.go.dev/math/cmplx@go1.18.1#Abs) function is used to get the modulus of the provided complex number. The modulus is calculated with `sqrt(x^2 + y^2)`, this gives the magnitude of the complex number. Here, we get the modulus as `9.43` as `sqrt(25 + 64)` for the complex number `5+8i`. Also, for `3+4i` the modulus becomes `sqrt(9+16)` which turns out to be `5`. The [Conjugate](https://pkg.go.dev/math/cmplx@go1.18.1#Conj) function is used to get the conjugate of the provided complex number. Also the phase or the Argument of the complex number can be obtained with the [Phase](https://pkg.go.dev/math/cmplx@go1.18.1#Phase) function. The phase is caluculated by the formula `tan^-1 (y/x)` but the angle is returned in randians. So for `x = 5+8i` the argument/Phase becomes `tan^-1( 8/5)` which is `57.995 degrees` or `1.012 radians`. We have the [Polar](https://pkg.go.dev/math/cmplx@go1.18.1#Polar) function which gives the polar form of the complex number i.e. `(modulus r, phase theta)` So this function returns two values the modulus and the argument/phase of the complex number. We have already calcualted both the values but this functions gets both of them in a single function. Quite neat, we can even ignore one value after the return of the function by using the ignore operator `_, phase := cmplx.Polar(5+7i)` to only care and get the phase/argument of the complex number or `modulus, _ := cmplx.Polar(5+7i)` to get the modulus from the complex number. So that's some basic operations on complex numbers, this might have very few use cases but it's still quite useful when needed. That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/mr-destructive/100-days-of-golang/) GitHub repository. ## Conclusion So from this section we were able to get a bit deeper introduction to the `math` package in golang's standard library. We covered some few important functions and constants in the main math package along with the glimpse of other subpackages like `rand`, `cmplx` and `bits`. We didn't get too much in detail with those sub packages as they can be explored on a separate section of their own. Hopefully, you have got a godd overview of the math package in golang which again is really important aspect in programming. Thank you for reading. If you have any questions or feedback, please let me know in the comments or on social handles. Happy Coding :) https://www.meetgor.com/posts/golang-packages Introduction In this 11th part of the series, we will be covering packages in golang. Package is a cool way to organize code in large projects. We can create a Fri, 22 Apr 2022 00:00:00 UTC ## Introduction In this 11th part of the series, we will be covering packages in golang. Package is a cool way to organize code in large projects. We can create a separate file which can include certain helper functions or variables from other files or scripts. There are couple of types of packages like packages from the standard library, open sourced community packages and custom packages that you can build of your own. In this particular section, we will be covering the fundamentals of packages and exploring the standard library in golang. ## What is a Package ? Package in simplest of terms is a collection of go scripts that can serve a purpose. Just like `fmt` has several functions like `Println`, `Printf`, `Scan`, `Scanf`, etc. most packages have functions that we can use in our own programs to solve a simple problem. We have already created many packages but none of them have been super useful so far, we just used the statement `package main` and didn't use the package anywhere. Every package has a entry point called `main` along with a entry function `main` which is triggered when we run the package. ``` - package_name - script_1.go - script_2.go - sub_package_name - utility_1.go - go.mod ``` The above is a simple folder structure for a package in golang, we have the package itself as the name of the folder. Inside of the package folder, we would have the scripts or sub-packages if any. Also there is a `go.mod` file in all go source package folders which contain the meta information about the package and it's dependencies. Let's take a look at the [fmt](https://github.com/golang/go/tree/master/src/fmt) package source code, it has a [print.go](https://github.com/golang/go/blob/master/src/fmt/print.go) file which has all the functions associated with printing, similarly separate files for different core functionality. We will soon into the details of packages in this series. Right now, we only care about the fundamentals of a package in golang. ## Importing Packages We have been already importing a package since writing our hello world in go, the `fmt` package which holds some functions for formatting, printing, logging and various string input/output manipulation. ```go import "fmt" ``` So, we have used the simple `import "package"` statement, but there are a couple of more ways to import packages if we have multiple packages to import. ```go import ( "fmt" "math" ) ``` Using the `()` and by specifying the name of the package we can import multiple packages at once. Also there is a harder way out there, if you really like to toil hard. ```go import "fmt" import "math" ``` This is generally avoided as it just looks too pythonic, we are in golang. ### Aliasing Imported packages We can alias an package a name whatever we want for the usage in the rest of the script file. This allows a bit better semantics of longer package names into readable code. ```go package main import ( "fmt" r "math/rand" ) func main() { fmt.Println(r.Int()) } ``` ``` $ go run import.go 3454565657742387432 ``` We have imported the package `math/rand`, here [rand](https://pkg.go.dev/math/rand@go1.18.1#Int) is a sub package of the main package math. Hence we have aliased the rand package as `r` and thus, we can use r to access all the functions and other types from the package. ### Blank Package Import We can even import the package but not use it without getting a compilation error. So, the blank identifier is used in golang to ignore the initialized or returned values from any context and avoid the compilation warning or errors. ```go package main import ( _ "fmt" ) func main() { } ``` ``` $ go run blank_import.go ``` Here, we have imported `fmt` package with a `_` blank identifier but we didn't use it anywhere, still we don't get any error messages and it compiled the code successfully. ## Standard Library Packages The Golang Standard Library has some good number of packages which can be used for some general tasks like formatting input/output, file handling, web requests, system commands, etc. You can check out the entire list along with the documentation on the [official website](https://pkg.go.dev/std). We can import these standard library packages just by parsing their name in the string quotes like we did with `fmt` as `"fmt"`. We have previously used the `rand` package from the math standard library package as a sub-package by using the statement `"math/rand"`, if we want we can import the entire `math` package as `"math"` but that's unwanted and we import only the package which we really need. There are other packages as well like `bufio` which is used for reading and performing operations with text, `os` for working with files systems and operating system level stuff, and other packages which are specific to use cases like rendering templates, time, sorting, math operations, encoding, etc. We will dive into some of them throughout this series. ## Installing Packages We can now get into installing other packages which are not in the standard library. You can get the documentation along with all references for a particular package on the official Golang [package repository](https://pkg.go.dev/). We use the CLI command to grab the packages into our `GOPATH`. OK, GOPATH, we have not covered this! ### GOPATH GOPATH is the path or the location in your system's disk where all the packages and modules are stored. You can get the default location of your GOPATH environment variable from the simple shell command. ``` $ echo $GOPATH C:\Users�cer\go ``` It has a few folders namely, `bin`, `pkg`, and `src`. These folder server different purpose like: - `bin` for storing the binaries generated from `go install` command - `pkg` for storing pre-compiled source files and objects for quicker generation of executables and compilation. - `src` for storing all the go source files of packages and modules. ### Go Get command Now, let's see how to install a package from the go community on GitHub. ``` go get github.com/gorilla/mux ``` We have installed a package which is a powerful HTTP router and a URL dispatcher and it can also be used to make web applications. It's called `mux`, we won't be using it right away just to get a feel for installing and playing with packages at the moment. After executing the command `go get` you should see a folder to be added in the `$GOPATH\pkg\mod` as `github.com\gorilla` and inside of it we should have a mux folder with the latest version. So, the `go get` command is used to download and install a package along with its all dependencies. ### Set up a project for using a package Now, we have got the package so we can import it from anywhere in our go environment. #### Create a new folder (any name) You can test a go package from a isolated environment from the GOPATH by creating using the mod command. The `mod init` command is a official way to create modules in golang and it creates kind of a environment to work on a templated project and structure the project/module/package properly. ``` go mod init ``` #### Install the packages We have already installed the package but that was a global install in the GOPATH, so we need to install it in this module. ``` go get github.com/gorilla/mux ``` #### Use the package Now, we can move into actually using the package in our source go file. We won't do any thing complicated just a simple web server. It's too easy don't worry! ```go package main import ( "net/http" "github.com/gorilla/mux" ) func main() { router := mux.NewRouter() router.HandleFunc("/", Server) http.ListenAndServe(":8000", router) } func Server(writer http.ResponseWriter, request *http.Request) { writer.Write([]byte("Hello Mux!")) } ``` We firstly setup a router(a pair of HTTP route with some logic) from the [NewRouter](https://pkg.go.dev/github.com/gorilla/mux#NewRouter) function provided by `mux`. We'll attach a function to this newly created router by pairing a URL with a function. So, in simple terms when the mentioned URL is visited or a GET request is sent(don't get into too much details) we want a function to be invoked or called which does something. Finally we will set up a web server that listens at a port with the created router. The final piece is the function which we will call when the URL is visited i.e. the `Server` function it can be any name. The function needs to have two arguments as it is invoked by a router, the writer and the request. The Writer is a Response writer i.e. to write the message to the server. We will simply use the [Write](https://pkg.go.dev/net/http#Header.Write) function to simply print a array of bytes. The type of the writer is specifically `http.ResponseWriter` as we want to write a simple HTTP response header. Also the request is a [http.Request](https://pkg.go.dev/net/http#Request) type as we simply accept a HTTP request. So, on running the following script, we will be able to see a simple HTTP response on the localhost at port `8000` or on your provided port. ``` go run main.go ```  That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/mr-destructive/100-days-of-golang/) GitHub repository. ## Conclusion So, we were able to dive a bit deeper in golang packages and modules. We covered from installation to importing packages in Golang, we also touched on basics of initializing a module in Golang. Hopefully, we were able to get the basics covered when it comes to packages in Golang. Thank you for reading. If you have any questions or feedback, please let me know in the comments or on social handles. Happy Coding :) https://www.meetgor.com/posts/golang-pointers Introduction In the tenth part of the series, we will be looking into an interesting concept in programming i.e. Pointer. It's a simple thing but a really power Tue, 19 Apr 2022 00:00:00 UTC ## Introduction In the tenth part of the series, we will be looking into an interesting concept in programming i.e. Pointer. It's a simple thing but a really powerful concept. Using pointers we can do several things very easily rather than writing a lot of code for a simple thing. We will be looking into basic concepts like declaration, referencing, de-referencing, and some examples on passing by reference, along with a pointer to struct instances. ## Pointers Pointers are simple, it's just their use case that makes it a big concept. Pointers are really powerful, they can do a lot of things that might seem impossible for a given problem. A pointer is a variable but unlike another variable which stores values in the form of integers, string, boolean, etc. pointers store the memory address. Memory address can be any valid location in memory that generally holds a variable. So, using pointers we can play with the memory address of variables and modify the contents of the variable directly using the memory address rather than accessing the variable. In golang, we have ways to store pointers and perform operations for the same. ## Declaring Pointers To declare pointers in golang, we can use the `*` before the type of data type we want to refer to. This means a pointer needs to specify which data type it is referencing as a measure of caution to mismatch types in the variable. Initially, the pointer variable is mapped to `<nil>` that is it points to nothing but a null pointer. ```go package main import "fmt" func main() { var ptr *int fmt.Println(ptr) } ``` ``` $ go run pointer.go <nil> ``` As we can see, the pointer that references an integer is initialized to nil. We have used `*` before the data type, this can be anything like `*string`, `*bool`, `*float64`, etc. ## The * and & in Pointers After declaring a pointer, we can now move into assigning a pointer a memory address. Using the `&` or the ampersand operator we can get the memory address of a variable. ```go var n := 34 var a_pointer *int = &n fmt.Println(a_pointer) ``` ``` $ go run pointer.go 0xc0000a6080 ``` Here, we can see that the pointer variable is storing the memory address of an integer variable. Don't worry about the value of the pointer variable, it is just a memory location on your machine. So, we use the `&` to access the memory address of any variable. We have seen that the `*` is used to declare a pointer variable, but it is also used for dereferencing a pointer. So, if we used `&` to get the memory address of a variable, similarly we can use the `*` to get back the value from the memory address. Both are opposite in terms of accessing the value. ```go n := 34 var a_pointer *int = &n fmt.Println(a_pointer) m := *a_pointer fmt.Println(m) ``` ``` $ go run pointer.go 0xc0000a8080 34 ``` As we can see, we have accessed the value stored in the pointer variable(`a_pointer`) by using the `*`. Here, the variable which we have created `m` will be of type whatever is stored in the memory address of the provided pointer variable. In this case, it is `int`, it can anything. So, this is how `*` and the `&` work in Golang. The `*` is used for declaring pointer variables as well as de-referencing pointer variables, and the `&` operator is used for accessing the memory address of the variable. That's basically the concept of pointers in golang. It's that simple. Using the simple concept of referencing and de-referencing, we can perform some operations like passing by reference to functions which will allow us to actually pass the value rather than the copy of the variable's value. ## Passing by Reference to Function Now we have the fundamentals of pointers cleared, we can move into actually using them to do some really useful operations. Generally, when we use parameters such as integers, strings, bool, etc. we are passing the copy of the variables into the function rather than the actual value of the variable. This is where pointers come in. By using pointers to pass the memory address of the variables we need to pass in we actually pass the location of the variables. Let's take a look at a simple example of a function that swaps the value of two variables. ```go package main import "fmt" func swap(x *int, y *int) { temp := *x *x = *y *y = temp } func main() { x := 3 y := 6 k := &x p := &y fmt.Printf("Before swapping : x = %d and y = %d. ", x, y) swap(k, p) fmt.Printf("After swapping : x = %d and y = %d. ", x, y) } ``` ``` $ go run pointer.go Before swapping : x = 3 and y = 6. After swapping : x = 6 and y = 3. ``` We can see here, that we have used pointers to pass the value of parameters to a function. Without using pointers, the value of the variable is passed as a copy but by using pointers, we are actually passing the memory address. In the main function, we first store the memory address of two variables `x` and `y` into two different pointer variables. We now can construct a function that accepts two memory addresses and perform further operations. Inside the function, we have de-referenced the pointer variables as with `*`. Don't confuse `x *int` with `*x`. We use `x *int` to make the function realize that we are passing a pointer variable of an integer value, and `*x` is used to de-reference the memory address which is stored in `x`. So, simply we - store the value in the memory location stored at `x` in the temp variable - store the value at the memory address stored in `y` into the memory address `x`. - store the value of the temp variable into the memory address stored in `x`. We have successfully swapped two values without returning any values from the function. ## Pointer to a Struct Instance/Object We can now even modify the values of Struct objects/instances by referencing the instance to a pointer. By assigning the pointer variable to a struct instance, we have access to its associated properties and function. Thereby we can modify the contents directly from the pointer variable. Let's take a look at a basic example of modifying properties using a pointer to a struct instance. ```go package main import "fmt" type Book struct { pages int genre string title string } func main() { new_book := Book{120, "fiction", "Harry Potter"} fmt.Println(new_book) fmt.Printf("Type of new_book -> %T ", new_book) book_ptr := &new_book book_ptr.title = "Games of Thrones" fmt.Println(new_book) } ``` ``` $ go run pointer.go {120 fiction Harry Potter} Type of new_book -> main.Book {120 fiction Games of Thrones} ``` So, we have created a pointer variable of the type which is a struct `Book`, this gives us access to the memory addresses associated with various properties defined in the struct. Using the pointer variable, we can access properties and thereby change the value directly as we have the memory address stored in `book_ptr`. So, if we say `book_ptr.title = "Games of Thrones"`, we are storing the string directly into the memory address of the `new_book` object as `book_ptr` refers to the memory addresses to the struct object `new_book`. Here, we have literally changed the value of a property in a struct object using pointers. This is really powerful and time-saving. If pointers were not a thing, you would have to write a separate function for doing the same. That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/mr-destructive/100-days-of-golang/) GitHub repository. ## Conclusion So, that's it we have covered enough basics of pointers so that we are able to understand the working of simple scripts or programs. Even simple data structures like slices or strings can be understood by using pointers in golang. From this part of the series, we were able to understand the declaration, referencing, and de-referencing of pointers along with passing by reference to functions and creating pointers to struct instances. Thank you for reading. If you have any questions or feedback, please let me know in the comments or on social handles. Happy Coding :) https://www.meetgor.com/posts/golang-structs Introduction Moving on to the 9th part of the series, we will be understanding structs in golang. Structs are an important aspect of programming in Golang, they Thu, 14 Apr 2022 00:00:00 UTC ## Introduction Moving on to the 9th part of the series, we will be understanding structs in golang. Structs are an important aspect of programming in Golang, they provide a way to define custom types and add functionality to them. We will be understanding the basics of operating on structs like declaration, initialization and adding functional logic into those structs. ## Structs in Golang Structs or Structures in Golang are the sequences or collections of built-in data types as a single type interface. Just like we have int, string, float, and complex, we can define our own data types in golang. They can consist of built-in data types as mentioned and also certain functions or methods which can be used to operate on them. Using structs we can create custom data types that can meet the specific requirements of our problem. We can define structs and later inside functions we can create instances of those structures. Structures are like a template or blueprint representation of data. It doesn't hold the actual data in memory, it is just used to construct an object of that type. After defining a struct, we can create instances or objects of those structs. These instances actually hold data in memory in the run time, so we basically deal with objects in the actual program. We'll see certain concepts of creating instances, declaring and defining structs, accessing data from instances and so on in the following section of the article. ``` Struct / Class Template / Structure for creating custom data types - Properties (variables and constants defined inside a structure) - Methods (functions that are bound to a struct) ``` ## Declaring Struct We can declare structs by using the keyword `type` followed by the name of the struct, after tha name, the `struct` keyword itself, and finally sets of parenthesis `{}`. Inside the parenthesis, we define the structure i.e. which type of data is to be stored and the name of those respective variables. ```go type Article struct { title string is_published bool words int } ``` We have declared a struct or a custom data-type or a class(not really) in golang with the name `Article` that has few associated properties/variables inside of it. We have `title` as a string, `is_published` as a boolean, and `words` as an integer value. This constructs a simple type of golang which has a defined structure. We can further use this Article struct as a data type in the main function or any appropriate scope for actually assigning the structure memory at runtime. ### Struct Naming Convention There are a few things that we need to understand and make a note of, especially the naming convention. - The struct name should be capitalized if you want to make it publicly accessible. - The variable/properties names i.e. `title`, `is_published`, and `words` should be also capitalized if you want to make them accessible from the struct instance. This might not be important right now but it is worth knowing for later use cases. Let's say we want to use a struct from other files or modules, for that the name of the struct in the file/script where the struct is defined should have the `Capitalized` convention. If you have a simple and single file script/program, you can keep it `lowercased` or `camelCased`. Leaving that aside, for now, we will try to focus on the essence of the structs in golang. ## Creating Instances/Objects of Structs Now, after defining the struct we need to create instances or objects of them. This can be done in several ways like using Struct literal, Manual assignment, and using the new function. We'll look into each of them in this section. ### Using struct literal The most simplest and straightforward way to initialize a struct is to use the struct literal just like we did with Maps, Slices, and Arrays. We basically parse the values of the respective fields in the struct. ```go package main import "fmt" type article struct { title string is_published bool words int } func main() { golang := article{"Golang Intro", true, 2000} fmt.Println(golang) } ``` ``` $ go run struct.go {Golang Intro true 2000} ``` We have created the object or instance of the struct `Article` using the shorthand notation or the walrus `:=` operator. Inside the `{}` braces, we can assign values but those values need to be in the same order as defined in the struct definition, else it gives a compilation error of `type mismatch`. So, here we have assigned the value `title`, `is_published`, and `word` as `Golang Intro`, `true`, and `2000` respective in that order. ### Using Key-value pairs We can also use the `key-value` notation for assigning values in the instance. With the previous method, we need to specify and thus initialize all the properties at once, but using this method we have a bit more flexibility. ```go vim := Article{title: "Vim: Keymapping", is_published: false} fmt.Println(vim) ``` ``` $ go run struct.go {Vim: Keymapping false 0} ``` Here, we have provided the key i.e. the variable name inside the struct, and then provided the value to it separated by a colon `:`. Using this way of initializing instances of struct we have better control and flexibility in providing a default value for that object. In the example above, we didn't initialize the property `words` but it already initialized to `0` since the object is created hence the memory allocation is completed, and thereby it needs to have a default value. ### Using the new function We can use the [new](https://pkg.go.dev/builtin#new) function to create a new instance of a struct. Though we can't provide an initial value, using the new function all the properties are initialized with their respective default values. Further, if we want to modify the values, we can access each property (variables in struct) using the `dot operator` and assign the desired values. ```go django := *new(Article) fmt.Println(django) ``` ``` $ go run struct.go { false 0} ``` We have used the new function to allocate memory for an instance of struct with the provided name. This function basically allocates all the properties of a default value and returns a pointer to that memory address. If we store the result of the new function in a variable object, we would get a pointer but we need the object itself, so we use `*` before the new function so as to de-reference the memory address from the pointer. So, we have stored the default values in the newly created object of Article structure in `django`, this gives the default values like an empty string `""`, default boolean value `false` and default integer value `0`. If we don't dereference the pointer and use it like `djagno := new(Article)`, thereby we get a pointer in that variable as `&{ false 0}`. Hence we use `*` before the new keyword. #### Accessing/Assigning values to properties We can now change the values of the properties in the object of the struct using the dot operator. We basically use the instance object name followed by a `.` and the property name to set its value. ```go django := *new(Article) fmt.Println(django) django.title = "Django View and URLs" django.words = 3500 django.is_published = true fmt.Println(django) ``` ``` $ go run struct.go { false 0} {Django View and URLs true 3500} ``` So, here we have used the object name which is `django`, and access any property by name with the `dot operator`, thereby we set the value as per the requirement. Note, we have not used the `:=` operator as the properties have already been initialized, we simply need to modify the default value. ## Creating Functions associated to Structs We can now move into creating functions in the struct, by adding functions/methods in structs we can incorporate a lot of functionality into the structure of our data type. For instance, we can set the value of a string as `"Empty"` or `"NA"` beforehand rather than empty string `""`. ```go package main import "fmt" type Mail struct { sender string subject string sent bool word_count int } func (m Mail) check_spam() { if m.subject == "" { fmt.Println("Spam!") } else { fmt.Println("Safe!") } } func main() { mail_one := *new(Mail) fmt.Printf("Mail one: ") mail_one.check_spam() mail_two := Mail{"xyz@xyz.com", "Golang Structs", true, 100} fmt.Printf("Mail two: ") mail_two.check_spam() } ``` ``` $ go run methods.go Mail one: Spam! Mail two: Safe! ``` We define a function associated with a struct by providing the `struct-name` and a parameter name which can be just used inside of the function. Here, we have used `(m Mail)` so as to reference the object of the struct provided to it. This basically binds the function to the struct and hence it becomes a method of that struct. Further, we can access the properties from the struct by their name using the dot separator. We are just checking whether the subject property in the instance is empty or not and simply printing text to the console. We are accessing the function and calling it with the syntax as `instance_name.function_name()`, here the function name is `check_spam` and the object name is `mail_one` for the first instance. Thereby we have called the function which is bounded to the instance of the struct. As we have accessed the function after the instance name the binding of the function i.e. the statements `(m Mail)` has taken the current instance and parsed it as the instance of the struct. Hence we are able to access the current instance's properties within the function/method. #### Adding a return statement By simply providing the return type and return statement with value, we can create functions of specific return types. ```go package main import "fmt" type Mail struct { sender string subject string sent bool word_count int } func (m Mail) check_spam() bool { if m.subject == "" { return true } else { return false } } func (m Mail) print_spam(spam bool) { if spam { fmt.Println("Spam!!") } else { fmt.Println("Safe!!") } } func main() { mail_one := *new(Mail) fmt.Printf("Mail one: ") is_mail_1_spam := mail_one.check_spam() mail_one.print_spam(is_mail_1_spam) mail_two := Mail{"xyz@xyz.com", "Golang Structs", true, 100} fmt.Printf("Mail two: ") is_mail_2_spam := mail_two.check_spam() mail_two.print_spam(is_mail_2_spam) } ``` ``` $ go run methods.go Mail one: Spam!! Mail two: Safe!! ``` We have modified the `check_spam` function which returns a boolean value. If the subject is empty it returns true else it returns false. Also, we have added a function `print_spam` function which takes in a parameter as a boolean value and prints text according to the value. This is how we work with functions in structs. We have parsed the return value of the `check_spam` function as a parameter to the `print_spam` function. ### Constructor in Structs Constructors are special methods that are invoked when the instance of a struct is created i.e. the properties are assigned an initial value or default value. In this way, we can perform basic operations which we need to perform after the instantiation of the struct. Golang does not have built-in constructors, but it is quite easy to create one. We simply need to create a function with an appropriate name(don't clash it with the struct name!!), by providing all the parameters that are in the struct so as to initialize them, and finally the return value as a reference to the struct instance. ```go package main import "fmt" type Repository struct { name string file_count int } func New_Repository(name string, file_count int) *Repository { file_count++ name = "Test" return &Repository{name, file_count} } func main() { blog := *New_Repository("", 0) fmt.Println(blog) } ``` ``` $ go run constructor.go {Test 1} ``` We have created a function that is technically acting like a constructor as it sets a default value to the properties in the structure. We have struct `Repository` containing `name` as a string and `file_count` as an integer. We created a Constructor function named `New_Repository` that basically takes in the properties in the struct, remember they haven't been initialized yet as we are writing the constructor for the very purpose. We have to parse the parameters with the initial value and let it modify once we have created the instance. That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/mr-destructive/100-days-of-golang/) GitHub repository. ## Conclusion So, from this part of the series, we are able to understand the basics of structs in golang. We covered declaration, definition, and adding methods in a struct. This gives a glimpse of Object-Oriented Programming in Golang. Thank you for reading. If you have any questions or feedback, please let me know in the comments or on social handles. Happy Coding :) https://www.meetgor.com/posts/golang-functions Introduction In the eighth part of the series, we will be exploring functions in golang. We will be diving into some basics of functions in golang like declarat Wed, 13 Apr 2022 00:00:00 UTC ## Introduction In the eighth part of the series, we will be exploring functions in golang. We will be diving into some basics of functions in golang like declaration, definition and calling. We won't be exploring all the topics of functions as it is quite a large topic to cover in one shot. So, building from the base, we will be starting from the basic declaration to simple return statements. ## Functions in Golang Functions in golang are a simple way to structure a block of code that can be re-usable. Functions also allow us to process a piece of logic and return the output. Functions allow us to write readable and scalable code as we have to write the code once and we can re-use the functionality of it by calling it. ## Declaring Functions We have already defined a function, if you have followed the series so far, or even written a `hello-world` application. The `main` function is the most fundamental function we can define in golang. The main is complicated if dive deeper but in the simplest of term it acts as a entry point for the entire program. ```go package main func main() { } ``` We have written the above code a lot of times till now, but never really talked about it's significance. Here we will understand the terminologies related to the main function. A function is declared with the `func` keyword along with the name of the function. There needs to be the `()` parenthesis after the name of the function, optionally it can take parameters inside the parameters to be used inside the function. We define the core functionality or the core logic of the function inside the braces `{}`. We also have the `return` keyword which can return values from the function to the block where we have called the function. Usually, we call a function from other function (most of the times it's the `main` function). The `return` keyword is not mandatory and it is usually added at the end of the function block, just before the closing braces `}`. ```go func hello_world_007() { } ``` We can define a custom function outside the main function by giving it a appropriate name. For the time bwing we can leave it empty and further define the logic of the actual function. The name of the function can be given as per the following standards: - Using letters`a-z A-Z`, numbers`0-9`, underscore `_` as a name. - Should not contain any spaces in-between the name. - Should not begin with a number or underscore. ## Defining Functions Inside the `{}` we define the actual functionality/logic of the function. The variables inside the function will remain local to the function and can't be accessed or altered from outside the function, though if we really want to access some global variables(from main or other functions) we can pass parameters, we will look into it in the next few sections. For time being, we will be focusing on the actual code block inside the function. ```go func hello_world() { fmt.Println("Hello World") } ``` This is a basic function that just calls another function `Println` from the fmt package, which basically prints text in the console. Though, we are using the function Println, it won't print the content to the string as we are not using/calling the function. Now, we can get a step ahead and start working with variables inside the function. ```go func hello_world() { name := "Gopher" fmt.Println("Hello", name) } ``` We have now added the local variable `name `inside the function, so this variable can only we referred inside the particular function. ## Calling Functions We can call the function from the main function or any other function by just specifying the name along with the `()` and optionally the parameters inside the parenthesis. ```go package main import "fmt" func main() { hello_world() } func hello_world() { name := "Gopher" fmt.Println("Hello", name) } ``` ``` $ go run func.go Hello Gopher ``` So, we define the function `hello_world` and call the function by using the statement `hello_world()` inside the main function and now, we are able to actually run the function. ## Passing Parameters We can optionally parse variables from a function to other and process it for further computation and programming. So, we can pass parameters in a function by specifying the name to be used inside the function followed by the type of that variable. ```go package main import "fmt" func main() { greet_me("Meet") n := "John" greet_me(n) } func greet_me(name string) { fmt.Println("Hello,", name, "!") } ``` ``` $ go run func.go Hello, Meet ! Hello, John ! ``` We have used the parameter `name` as a string in the function and used it inside the function body. The parameter name which is to be called from the main function can be anything and not necessarily be the same as declared in the function declaration. For instance, we have used the variable in the main function `n` which is passed in the function call. We can even pass the value as it is to the function in golang. ## Return Keyword We can use the return keyword to actually return a value from the function and not just display the message. The returned value can be later used from other places in the program. ```go package main import "fmt" func main() { // return value y := line_eq(3, 1, 2) fmt.println("for x = 3 , y = ", y) } func line_eq(x int, m int, c int) int { return ((m * x) + c) } ``` ``` $ go run func.go for x = 3 , y = 5 ``` So, here we are able to fetch the returned value from the function and store it in another variable and further compute the required logic. We also need to specify the return type of the function after the parameters like `func (parameters) return-type { }`. Here, we need to return the specified type of the return value from the function else it would give a compilation error. So, we basically need to provide the return value and also the return statement to capture the value from the function call. ### Multiple return values We can also provide multiple return values by providing a list of return values like `(type1 type2 type3 ....)`. We can return the values by separating the values by a comma. So, while calling the function, we need to specify the variables again as comma-separated name and this will capture the value from the function call. ```go package main import "fmt" func main() { // multiple return values s, c, odd := sqube(5) fmt.Println("for x = 5 , x^2 =", s, "x^3 =", c) if odd == true { fmt.Println("x is odd") } else { fmt.Println("x is true") } } func sqube(x int) (int, int, bool) { square := x * x cube := x * x * x var is_odd bool if x%2 == 0 { is_odd = false } else { is_odd = true } return square, cube, is_odd } ``` ``` $ go run func.go for x = 5 , x^2 = 25 x^3 = 125 x is odd ``` So, we have returned multiple values from the function like two integers and one boolean. The parameter is a single integer, now we need to parse 3 variables in order to capture all the values from the function call. Thus, we are able to get all the values from the function. That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/mr-destructive/100-days-of-golang/) GitHub repository. ## Conclusion So, from this part of the series, we are able to understand the basics of functions in golang. We covered from declaration, definition and simple return statements and function calling. Thank you for reading. If you have any questions or feedback, please let me know in the comments or on social handles. Happy Coding :) https://www.meetgor.com/posts/golang-maps Introduction In the seventh part of the series, we will be covering Maps. We have covered some basic data structures like arrays and slices, and now we can move Tue, 12 Apr 2022 00:00:00 UTC ## Introduction In the seventh part of the series, we will be covering Maps. We have covered some basic data structures like arrays and slices, and now we can move into maps or hash tables. Maps allow us to store key-value pairs of a particular type. In this part of the series, we will be covering the basics of Maps in Golang like declaration, iteration, and Creating, updating, and deleting keys from the map. ## Maps in Golang Maps in golang are data structures that provide a way to store key-value pairs. It is also known as a hash table. Maps allow us to create unique keys which are associated with a value. It can be used to create a data structure that can have an item that is associated with a particular value, for example, the basic example of the map can be a frequency table of a list of numbers. We can store the frequency of each element occurring in the list. Let's say we have a list of numbers as `[3, 5, 9, 4, 9, 5, 5]`, we can create a map of the frequency of these elements as `[3:1, 5:3, 4:1, 9:2]`. Here, we have stored the information in the form of `key-value` pairs as a frequency. So, `3` has occurred one time, `5` 3 times, and so on. Maps are not stored in order of the numbers they are unordered so we need to manually sort them in the order we want. ## Declaring Maps We can declare maps by defining the type of mapping like the two types we are mapping. We can map any type with any other, like a character with an integer, an integer with an integer as we saw earlier, etc. We have several ways to decalre maps in golang, like using map literal, make function, new function, and a few others. We'll look into each of them in a brief. ### Simple map literal As we saw in the array and slices, we used the slice literals to declare and initialize an array or a slice. Similarly, we can use the map literal to create a map in golang. Here, we use the `map` keyword followed by the two types of data we are going to map with. ```go package main import "fmt" func main() { char_freq := map[string]int{ "M": 1, "e": 2, "t": 1, } fmt.Println(char_freq) } ``` ``` $ go run map.go map[M:1 e:2 t:1] ``` We have used the map keyword to initialize a map with a `string` with `int`. The first data type is declared inside the square brackets`[]` and the second data type outside the square brackets. We use the `{}` to define the map values. We can even leave the `{}` empty. ``` char_freq := map[string]int{} ``` We initialize the values of the map by specifying the data for that data type in this example a string `""` followed by a colon `:` and finally the value of the second pair data. Each value is separated by a comma(`,`). ### Using make function We can even use the [make](https://pkg.go.dev/builtin#make) function to create a map in golang. The make function is used for allocating memory. The make function allocates memory which might be enough for the initial values provided. It allocates more memory as the map grows in size. We use the make function by providing the `map` keyword along with the data types of the key values pairs to be mapped. Optionally we can provide the capacity as we provided in the slice declaration. It basically doubles once it reaches the limit and is re-allocated. ```go marks := make(map[int]int) marks[65] = 8 marks[95] = 3 marks[80] = 5 fmt.Println(marks) ``` ``` $ go run map.go map[65:8 80:5 95:3] ``` We have used the `make` function for declaring the map, the initial size is around 7 if not mentioned. After it hits 7, the capacity is mostly doubled and increased as per the modifications. ### Using the new function We can even use the [new](https://pkg.go.dev/builtin#new) function(a bit hacky) to crated a map in golang. The new function basically is used to allocate memory but is not the same as the `make` function, it returns the memory address to an allocated pointer. So, we can set the value of the returned function call of the new function with a pointer variable. A pointer in golang is simply a reference to a memory address, we'll dive into pointers in a different section. After the pointer is assigned a memory address, we can refer to the address of that pointer and thus access the original value which is the map itself. ```go name := new(map[byte]int) *name = map[byte]int{} name_map := *name name_map['m'] = 1 name_map['e'] = 2 name_map['t'] = 1 fmt.Println(name_map) for k, _ := range name_map { fmt.Printf("%c ", k) } ``` ``` $ go run map.go map[101:2 109:1 116:1] m e t ``` So, we can see we created the map with the new function and stored the address into a pointer, later we initialized the empty map and stored the initial reference in the same pointer address. Then, we can finally store the map in another variable so that we can use it as a normal map. So, this is how we declare the map using the new function. ## Access Keys and Values in Maps We can access the values by simply accessing them with the keys. Using the square bracket and the key literal into the braces, we get the value associated with that key. For example, the map `["M": 1, "E": 2, "T":1]`, we can use the `map_name["E"]` which will get the value as `3`. ### Length of Map The length of the map can be accessed using the len function, the len function returns the number of key-value pairs in the map. ```go char_freq := map[string]int{ "M": 1, "e": 2, "t": 1, } fmt.Println(char_freq) fmt.Println(len(char_freq)) ``` ``` $ go run map.go map[M:1 e:2 t:1] 3 ``` ## Check for existing Keys in Map We can check if a key exists in the map by using the comma-ok syntax. The key can be accessed using the first variable and if the key doesn't exist, the second variable is set to false. So, we can verify the existence of a key in the map using the two-variable approach. ```go name_map := map[byte]int{ 'm': 1, 'e': 2, 't': 1, } var key byte = 't' value, exist := name_map[key] if exist == true { fmt.Printf("The key %c exist and has value %d ", key, value) } else { fmt.Printf("The key %c does not exist. ", key) } ``` ``` $ go run map.go The key t exist and has value 1 ``` So, we can see the exist value is true if the key exists and false if it doesn't. So, we can then verify if a particular key exists in a map or not. ## Adding and Modifying Keys/Values in Maps We can add a key-value pair in a map by just using the key as we did in the initialization process. We simply pass the key in the square braces `[]` and assign it a value appropriate to the data type used in the map. ```go cart_list := map[string]int{ "shirt": 2, "mug": 4, "shoes": 3, } fmt.Println(cart_list) cart_list["jeans"] = 1 cart_list["mug"] = 3 fmt.Println(cart_list) ``` ``` $ go run map.go map[mug:4 shirt:2 shoes:3] map[jeans:1 mug:3 shirt:2 shoes:3] ``` We can access the keys in the map by just using the key as it is and altering the value it holds, the same thing applies to the addition of the key-value pairs, we can use the key and assign the value associated with it. ## Delete Keys in Maps We can delete the key-value pairs in the map, using the `delete` function. We pass in the `key` and the map to delete the key-value pair from the map. ```go cart_list := map[string]int{ "shirt": 2, "mug": 4, "shoes": 3, } fmt.Println(cart_list) cart_list["jeans"] = 1 cart_list["mug"] = 3 delete(cart_list, "shoes") fmt.Println(cart_list) ``` ``` $ go run map.go map[mug:4 shirt:2 shoes:3] map[jeans:1 mug:3 shirt:2] ``` So, we can see the key-value pair was deleted from the map. ## Iterate over a Map We can iterate over a map similar to the range keyword iteration for slices and arrays, but the exception here, is that we use the key, value instead of the index, copy of an element in the map as the range. ```go is_prime := map[int]bool{ 7: true, 9: false, 13: true, 15: false, 16: false, } for key, value := range is_prime { fmt.printf("%d -> %t ", key, value) } ``` ``` $ go run map.go 9 -> false 13 -> true 15 -> false 16 -> false 7 -> true ``` So, we can observe that we can access the keys and values in the map using the range keyword for iterating over the map. Inside the for loop, we can refer to the assigned values present in the map. ### Use only key or value while iterating If we don't use either of the variables like `key` or `value`, the compiler might give us the unused variable error, so we have an alternative to use don't care variables namely the `_` underscore character. ```go is_prime := map[int]bool{ 7: true, 9: false, 13: true, 15: false, 16: false, } for key, _ := range is_prime { fmt.Printf("Key : %d ", key) } for _, value := range is_prime { fmt.Printf("Value: %t ", value) } ``` ``` $ go run map.go Key : 7 Key : 9 Key : 13 Key : 15 Key : 16 Value: true Value: false Value: true Value: false Value: false ``` So, we use the `_` to ignore the usage of the variable in the loop, if we are not sure of using any variable, we can ignore it completely with the underscore operator and thus prevent any compilation errors/warnings. So, here if we want to only access keys, we use `key, _` in order to fetch only keys and silence the values in the map. If we want to access only the values, we can use `_, value` so as to get all the values from the map. The variable name `key` or `value` can be anything but make sure to use those only inside the loop. That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/mr-destructive/100-days-of-golang/) GitHub repository. ## Conclusion So, from this part of the series, we were able to understand the basics of maps in golang. We covered some basics stuff including the declaration, initialization, and iteration. Maps are quite simple but important for creating interesting applications. Thank you for reading. If you have any questions or feedback, please let me know in the comments or on social handles. Happy Coding :) https://www.meetgor.com/posts/golang-slices Introduction In the sixth part of the series, we will be covering slices. Slices are almost like arrays but have a lot of advantages over them, including flexib Thu, 07 Apr 2022 00:00:00 UTC ## Introduction In the sixth part of the series, we will be covering slices. Slices are almost like arrays but have a lot of advantages over them, including flexibility and control over them. We can adjust the size and capacity of the data which we will store at a place using slices. We will be covering basic declaration, initialization, capacity in slices, iteration, and accessing the elements of the slices. ## Slices in Golang Slices are Arrays but can provide more control and are more flexible than arrays. In slices, we can increase the size of the array/list of elements whenever required. We can even have a capacity for slices i.e. the maximum length we wish to grow the initial slice. Though slices are dynamic, it has a few disadvantages like compile safety, access time, comparability, etc. Everything has its pros and cons, you have to decide on the right data structure as per your problem statement and requirements. ## Declaring Slices There are a couple of different ways in which we can declare a slice that might be an uninitialized or initialized slice. Some of the standard ways include using the `make` function and the normal array-like declaration though there are other methods as well including using the `new` function. We'll explore some of the most easiest and handy ways to declare and initialize slices. ### Using array-like declaration Slices can be declared quite similar to arrays but we don't enter the initial size(length). As discussed in the array part, we can use the var keyword and the square brackets. Though you don't have to enter the length inside the `[]` brackets, the type of the slice needs to enter. ```go package main import "fmt" func main() { var marks [] int fmt.Println(marks) } ``` ``` $ go run slices.go [] ``` So, initially, it is empty without an element as we have not initialized any elements. If you enter the length in the `[]` brackets, it would be an array and not a slice. We can also initialize the elements in the slice with the slice literal as we did with arrays using an array literal. ```go frameworks = []string{"Django", "Laravel", "Flask", "Rails"} fmt.Println(frameworks) ``` ``` $ go run slices.go [Django Laravel Flask Rails] ``` ### Using make function We can even use the [make](https://pkg.go.dev/builtin#make) function to create a slice in golang. The make function basically allows us to create a slice by providing the length as well as the capacity. Let's clear the concept of Capacity and Slice first. #### Capacity in Slice Capacity in slices is the initial value provided during the declaration of a slice. It basically holds the capacity of the slice to grow beyond the length. OK, let's take an example, If you declare a slice using make with length 3 and capacity 5, you initially can access 3 elements but memory has been allocated for 5 elements, if your slice exceeds this capacity it will double its original capacity. #### Obtain Length and Capacity of Slice We can get the capacity by using the [cap](https://pkg.go.dev/builtin#cap) function just like the `len` function. In slices, we can use the lens function to get the length and the cap function to get its underlying capacity to grow. It's just a way for us to manage the memory reallocation for a slice. Slices under the hood are arrays with a more dynamic punch. So, now we can a bit confidently move to the make function for declaring slices. ### Back to make function The Make function is used to declare and initialize a slice (not only slice but maps and channels as well). The function primarily takes in 3 parameters namely, the type of slice, the initial length of the slice, and optionally the capacity of the slice. If we don't provide the capacity, the capacity is set the same as the length. ```go var langs = make([]string, 3, 5) langs[0], langs[1], langs[2] = "Python", "Go", "Javascript" fmt.Println(langs) fmt.Printf("Length = %d Capacity = %d ", len(langs), cap(langs)) langs = append(langs, "Java", "Kotlin", "PHP") fmt.Println(langs) fmt.Printf("Length = %d Capacity = %d ", len(langs), cap(langs)) ``` ``` $ go run slices.go [Python Go Javascript] Length = 3 Capacity = 5 [Python Go Javascript Java Kotlin PHP] Length = 6 Capacity = 10 ``` So, there are a lot of things to take in here. We use the make function by parsing in three parameters as said the type in the form of `[]type` as a distinction for creating slices because we also use the map to create maps and channels. The next two parameters are length and capacity. So, we have initialized a slice of type string, length 3 i.e. we are saying we will initially access only three elements from the array, finally, the third parameter is the capacity which will be the already initialized array under the hood for the slice. So, we have already created an array(under the hood) with 5 elements initialized but only 3 accessible from the slice interface. Further, we initialize/modify the elements in the created slice. We set 3 elements to some strings and that way we have all elements filled with non-default values in the slice. Now if we say `langs[3] = "Something"` it would give an error like `panic: runtime error: index out of range [3] with length 3`. This is a panic in golang which can be triggered in one of the ways in the slice when you access an unreferenced element in it. We have initialized the element but not in the slice interface. So, we have a particular function called append which appends and thus grows the length beyond its current length and refers to the elements initialized from the capacity. So initially the capacity was 5 and after adding the 6th element, it doubled down to `10`. So, under the hood, all 5 elements in the array would have been re-allocated into a new memory location and the sixth element would have been added to the next location. This is how we efficiently re-allocate memory for elements in slices. We have a bit more control over the length and are more flexible than arrays. ### Using new function The [new](https://pkg.go.dev/builtin#new) function in golang is used to allocate a slice/any other type. We can use the new function so as to mimic the make function by adding a bit of value like the initial capacity and length. You can refer to the [article](https://www.golangprograms.com/go-language/slices-in-golang-programming.html) for providing the original idea. ```go langs2 := new([3]string)[0:2] langs2[0], langs2[1] = "Python", "Go" fmt.Println(langs2) fmt.Printf("Length = %d Capacity = %d ", len(langs2), cap(langs2)) langs2 = append(langs2, "Java", "Kotlin", "PHP") fmt.Println(langs2) fmt.Printf("Length = %d Capacity = %d ", len(langs2), cap(langs2)) ``` ``` $ go run slices.go Length = 2 Capacity = 3 [Python Go Java Kotlin PHP] Length = 5 Capacity = 6 ``` So, it would work almost similar to the make function. But by default, it would work as nil if you don't provide any length in the `[]` brackets like `new([]string)`. This will create a empty slice `[]` with zero capacity and zero length. ## Adding elements in Slice We had a few spoilers for this already with the `append` function. The [append](https://pkg.go.dev/builtin#append) takes in the variable and then the list of values that we want to add. Here, if the capacity of the slice is exceeded, it re-allocates the slice to a new location, and the elements are moved to that location and then the provided elements are added. ```go var percentages = []float64{78.8, 85.7, 94.4, 79.8} fmt.Println(percentages) percentages = append(percentages, 60.5, 75.6) fmt.Println(percentages) ``` ``` $ go run slices.go [78.8 85.7 94.4 79.8] [78.8 85.7 94.4 79.8 60.5 75.6] ``` So, here we can see the append function adding the elements in a slice. You can pass as many elements(MaxInt i.e. int64 or int32 elements precisely) you require in the append function. Calling the append function, again and again, might degrade the efficiency though, so make sure to add a right number of elements in a single call. ## Accessing and Modifying elements in Slice We can simply access the elements using the index in the `[]` brackets. But there is more to that in slices. We can actually get slices of slices. Even in arrays or slices, we can get the particular elements between a specific two indices like 2 and 4, so we can write `array[2:4]` to get elements at index `2`, and `3` the upper bound is non-inclusive. But if we want to have all the elements from a specific index to the last element, we can leave the number blank as `[2:]` would give elements from index 2 to the last index in the array/slice. ```go scores := []int{80, 85, 90, 75, 60, 56, 83} fmt.Println(scores) fmt.Println("From index 2 to 4", scores[2:5]) fmt.Println("From index 0 to 2", scores[:3]) fmt.Println("From index 3 to 5", scores[3:]) ``` ``` $ go run slices.go [80 85 90 75 60 56 83] From index 2 to 4 [90 75 60] From index 0 to 2 [80 85 90] From index 3 to 5 [75 60 56 83] ``` So, we are able to perform index slicing in golang on arrays and slices. Further, as for the array, we can also modify elements in slices. Using the index of that element, we can access the element and perform operations on it and thus change the literal value of the element. ```go word := []byte{'f', 'u', 'z', 'z', 'y'} fmt.Printf("%s ", word) word[0] = 'b' word[len(word)-1] = 'z' fmt.Printf("%s ", word) ``` ``` fuzzy buzzz ``` So, now we can also modify existing values of elements in slices. ## Deleting elements from Slice We can also remove an element from the slice i.e. shrink the length of the slice. There is no function to remove an element from the slice, but we can work around with the append function in golang. So, in the slice before the element's index to be deleted is appended with all the elements after the index of the element to be deleted. ``` 10 20 30 40 50 60 0 1 2 3 4 5 Delete the element at index -> 2 Copy from 3 to 5 into the slice from 0 to 1 // append(slice[:2], slice[2+1:] ^ ^ | | 10 20 40 50 60 append 40 50 60 -> 10 20 10 20 40 50 60 ``` Here in the above example, we append the indices after the element to be deleted into the slice of elements before the `to be deleted element`. ```go marklist := []int{80, 85, 90, 75, 60} fmt.Println(marklist) var index int fmt.Printf("Enter the index to be deleted: ") fmt.Scan(&index) elem := marklist[index] // append in such a way that the element to be removed is excluded marklist = append(marklist[:index], marklist[index+1:]...) fmt.Printf("The element %d was deleted. ", elem) fmt.Println(marklist) ``` ``` $ go run slices.go [80 85 90 75 60] Enter the index to be deleted: 3 The element 75 was deleted. [80 85 90 60] ``` ## Iterate through a slice As arrays are under the hood modifications of arrays, we have a quite similar approach to iterating over slices in golang. ### Using three statements for loop We can use the three statements for loop i.e. the initialization, condition, and incrementation procedure. The counter is set from 0 or any other starting value as the index of the slice, next we have the end loop condition i.e. a condition to check until when to exit, and finally the amount to which we need to increment the counter. ```go code := [7]rune{'g', 'o', 'l', 'a', 'n', 'g'} for i := 0; i < len(code); i++ { fmt.Printf("%c ", code[i]) } ``` ``` $ go run slices.go g o l a n g ``` ### Using Range-based for loop We can use range-based for loops to iterate over the slice elements. The range keyword is passed with the slice name or the slice of an array to iterate over. Using the two variables i.e. the iterator and the copy of the element, we can access the index and the element in the slice. ```go scores := []int{80, 85, 90, 75, 60, 56, 83} for _, s := range scores { fmt.Println(s) } ``` ``` $ go run slices.go 80 85 90 75 60 56 83 ``` We can also use a slice of slice i.e. scores[:4] to access a specific element in the range using index slicing. ```go scores := []int{80, 85, 90, 75, 60, 56, 83} for _, s := range scores[1:4] { fmt.Println(s) } ``` ``` $ go run slices.go 85 90 75 ``` ### Using for loop with range We can even use them for loop as a while loop to look and get a bit of both the above methods like the range method and the typical for loop access. ```go start, i, end := 2, 2, 5 modes := []string{"normal", "command", "insert", "visual", "select", "replace"} for range scores[start:end] { fmt.Printf("Element at index %d = %s ", i, modes[i]) i++ } ``` ``` $ go run slices.go Element at index 2 = insert Element at index 3 = visual Element at index 4 = select ``` So by using the range keyword we were able to iterate over the slice but without assigning the iterator and the copy of the element, we manually set a counter `i` and increment it as per our liking. Using index slicing we were able to get the elements between particular indices. That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/mr-destructive/100-days-of-golang/) GitHub repository. ## Conclusion So, from this part of the series, we were able to understand the basics of slices in golang. We covered some basics stuff including the declaration, initialization, and iteration. We also covered the under the hood working of slices and how to relate with the arrays. Thank you for reading. If you have any questions or feedback, please let me know in the comments or on social handles. Happy Coding :) https://www.meetgor.com/posts/golang-arrays Introduction In this fifth section of Golang, we will be understanding the basics of arrays. We will be covering some basic operations on arrays like declaratio Wed, 06 Apr 2022 00:00:00 UTC ## Introduction In this fifth section of Golang, we will be understanding the basics of arrays. We will be covering some basic operations on arrays like declaration, initialization, modifications, and iterating over an array. ## Declaring Arrays Arrays are type of data structure that allow us to store multiple items at continuous memory locations of the same type. In golang, we can create arrays similar to any variable but by adding a few bits and pieces like the `[]` square braces, length of the array, values, etc. In golang, we cannot resize the length once it is initialized. To create a basic array in golang, we can use the following code: ```go package main import "fmt" func main() { var languages[4]string languages[0] = "Python" fmt.Println(languages) } ``` ``` $ go run basic.go [Python ] ``` Here, we have initialized a string array of size 4 and not initialized any values of the elements in the declaration. We later set the value of the 0th index or the first element in the array to a string and still rest of the elements are by default set to empty strings `""`. The point ot be noted here, is that the size of the array cannot be changed later nor you can leave the size empty while declaring an array in Golang. We can even initialize elements at the time of declaring the array as follows: ```go lang_array := [4]string {"Python", "Go", "Javascript", "C++"} fmt.Println(lang_array) ``` ``` $ go run basic.go [Python Go Javascript C++] ``` So, we use the walrus operator `:=` to initialize an array with the values at the time of declaration. ### Letting Compiler makeout the array length We can even let the compiler decide the length of the array, using the `...` syntax inside the brackets. This is limited by using the array literal that is by initializing values in the `{}` braces. So, all the elements need to be declared in the array. ```go cart := [...]string {"Bag", "Shirt", "Watch", "Book"} fmt.Println(cart) ``` ``` $ go run basic.go [Bag Shirt Watch Book] ``` ## Access and Modify Elements To access an element in the array, we can use the index of that element which starts from 0 as usual in programming. ```go marks := [6]int {85, 89, 75, 93, 98, 60} fmt.Println(marks[1]) fmt.Println(marks[5]) fmt.Println(marks[3]) ``` ``` $ go run basic.go 89 60 93 ``` We can now, access the element at a particular index in the array. Now, we will see how to modify or edit the elements which are already initialized. ```go name := [5]byte {'f','u','z','z','y'} fmt.Printf("%s ",name) name[0] = 'b' name[4] = 'z' fmt.Printf("%s ",name) ``` ``` $ go run basic.go fuzzy buzzz ``` By accessing the index of the element we can set a appropriate value to the element in the array and thus we have modified the contents of the array. ## Find Length of Array To find the length of the Array, we have the `len` function. The [len](https://pkg.go.dev/builtin#len) function takes in the array as the parameter and returns the size of the array(int). ```go code := [7]rune {'#', '5', 'g', 't', 'm', 'y', '6'} fmt.Println("The length of the array is :", len(code)) ``` ``` $ go run basic.go The length of the array is : 7 ``` In the previous few section, we talked about letting the compiler make out the length of the array while declaring and initializing the array, we can use the len function to calculate the length of the array for further computation ```go cart := [...]string {"Bag", "Shirt", "Watch", "Book"} fmt.Printf("There are %d items in your cart ", len(cart)) ``` ``` $ go run basic.go There are 4 items in your cart ``` We can now get the length of the arrays even with `[...]` syntax using the len function. ## Iterate over an Array We can move on to the most important aspect when it comes to arrays i.e. to iterate over each element. We can use various types of for loops like the three statement for loops, range based loop or while loop. ### Three statement for loop We can use the three statement for loop, the initialization statement as to `0`, condition to be the counter (i) should be less than the length of the array by using the `len` function and increment each time by one. ```go code := [7]rune {'#', '5', 'g', 't', 'm', 'y', '6'} for i := 0; i<len(code); i++{ fmt.Printf("%c ",code[i]) } ``` ``` $ go run basic.go # 5 g t m y 6 ``` Thus, we can iterate over the array with three statement for loop in golang. ### Using range-based loop We can use the `range` keyword to iterate over the arrays in golang. The range keyword is used to iterate over the array by taking two variables i.e. the iterator and the copy of the element in the iterator. We don't have any use of the iterator so we say it as `_`, otherwise it gives a warning/error of not using declared variables. So, we only require the copy of the element in this case, so sayit as `s` or any other name you like. ```go cart := [...]string {"Bag", "Shirt", "Watch", "Book"} for _, s := range cart{ fmt.Println(s) } ``` ``` $ go run basic.go Bag Shirt Watch Book ``` Thus, using the range based for loops we were able to iterate over the array for each element without needing any check condition and incrementation of the counter/iterator. That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/mr-destructive/100-days-of-golang/) GitHub repository. ## Conclusion So, from this part of the series, we were able to understand the baiscs of arrays in golang. We covered from declaration of arrays to iteration. Thank you for reading. If you have any questions or feedback, please let me know in the comments or on social handles. Happy Coding :) https://www.meetgor.com/posts/golang-input Introduction In this fourth section of Golang, we will be understanding the basics of user input. In golang, we can get user input with several functions most o Tue, 05 Apr 2022 00:00:00 UTC ## Introduction In this fourth section of Golang, we will be understanding the basics of user input. In golang, we can get user input with several functions most of which are similar to the `C` programming language like `scanf`. This type of input is quite powerful and gives more control on the input to be received. ## Scan Function The [Scan](https://pkg.go.dev/fmt#Scan) function helps in getting a value with space as delimiter i.e. The input is stored before a space is encountered. This means the input is only limited to adding a space or a new line. We can use the function by passing the reference to the variable we are going to store the input value. So, we can have a basic input in Golang as follows: ```go package main import "fmt" func main() { var pname string fmt.Println("Enter your favourite programming language: ") fmt.Scan(&pname) fmt.Println("So, your favourite programming language is",pname) } ``` ``` $ go run scan.go Enter your favorite programming language: python So, your favorite programming language is python ``` We need to declare the variable to take input as we need a reference of that variable to store the input. We will be talking about `&` and pointers in a separate article. We use the `Scan` function by passing the reference to the variable `pname` like `&pname` which means, fetch the memory address of the variable `name`, we just pass the address as `int` to the Scan function and it does the rest to store the input value in it. We then as usual access the variable and operations on it. Here, if you add a space in the input, the value after the space won't be picked by the Scan function. It strictly stops accepting values input after it sees space. We can use this to input multiple variables at once. We know scan gets input before encountering space, so we can pass multiple variable references and add them as input. ```go var ( name string age int gender rune ) fmt.Println("Enter your name age and gender: ") fmt.Scan(&name, &age, &gender) fmt.Printf("Hello %s, you are a %c and %d years old", name, gender, age) ``` ``` $ go run scan.go Enter your name age and gender: Meet 19 77 Hello Meet, you are a M and 19 years old ``` Here, we are declaring multiple variables like `name`, `age`, and `gender` as `string`, `int`, and `rune` respectively. Then, we can input all of these in a single scan statement by comma-separated variables. Here, we need to input the `rune` as an int value because under the hood it is an integer alias. So, we inputted `77` which is equivalent to `M` in ASCII characters and even Unicode character sets. Thus, we were able to input multiple variables with the Scan function. ## Scanf functions The [Scanf](https://pkg.go.dev/fmt#Scanf) function is quite similar to the `scanf` in C programming language as it allows to specify the type of the incoming input. This will solve the problem of us inputting `77` instead of `M` in the gender variable in the previous example. The Scanf function allows us to take input by specifying the placeholder types and the delimiters as well. The delimiter is basically the separator between two or more entities. We can either use space separation or ` ` as an input delimiter i.e. the way we want to separate inputs from each other while taking input. ```go var ( name string age int gender rune ) fmt.Println("Enter your name age and gender: ") fmt.Scanf("%s %d %c", &name, &age, &gender) fmt.Printf("Hello %s, you are a %c and %d years old", name, gender, age) ``` ``` $ go run scanf.go Enter your name age and gender: Meet 12 M Hello Meet, you are a M and 12 years old ``` How cool is that? It definitely gives much more control on what and how to take input. We are taking input as only space-separated values. Let's now try to get more control over how the input will be taken and stored. ```go var ( name string age int gender rune ) fmt.Println("Enter your name age and gender: ") fmt.Scanf("%s %d %c", &name, &age, &gender) fmt.Printf("Hello %s, you are a %c and %d years old", name, gender, age) ``` ``` $ go run scanf.go Enter your name age and gender: Meet 12 M Hello Meet, you are a M and 12 years old ``` By adding ` ` between the `%s`(name) and `%d`(age), we want the user to type the name on one line and age with gender on a different line. The age and gender as before separated by space. ## Scanln function The [Scanln](https://pkg.go.dev/fmt#Scanln) function is a modification of the Scan function as it only stops the input after a newline/enter is pressed. So, using this we can input multiple variables which are space-separated in a single line. ```go var s string fmt.Println("Enter a string: ") fmt.Scanln(&s) fmt.Println(s) ``` ``` $ go run scanln.go Enter a string: $ go run scanln.go Enter a string: Can't type Can't $ ype -bash: ype: command not found ``` The Scanln function even accepts an empty string as input. It just needs to get the new line character and it will exit, it also only accepts space-separated values. The rest of the input after space is thrown away and is basically exited from the program stream. More specifically, the input `Can't Type` was treated only as `Can't` anything after the space is not considered in the input value. The key difference between Scan and Scanln is that Scanln will not accept input that is space-separated, Scan function considers the newline/enter as a space if there are multiple inputs. The below example will make things absolutely clear. ```go // scan.go package main import "fmt" func main() { var ( name string age int gender rune ) fmt.Println("Enter your name age and gender: ") fmt.Scan(&name, &age, &gender) fmt.Printf("Hello %s, you are a %c and %d years old", name, gender, age) } ``` ```go //scanln.go package main import "fmt" func main() { var s string fmt.Println("Enter a string: ") fmt.Scanln(&s) fmt.Println("Inputted string : ", s) } ``` ``` $ go run scan.go Enter your name age and gender: Meet 14 77 Hello Meet, you are a M and 14 years old $ go run scanln.go Enter a string: Inputted string : ``` We can see that, The Scan function won't exit until it has inputted all its input values even with newline and spaces. Whereas the Scanln function just waits for the newline character (Enter Key) to be pressed and it exits, thereby even allowing an empty string as input. That's it from this part. Reference for all the code examples and commands can be found in the [100 days of Golang](https://github.com/mr-destructive/100-days-of-golang/) GitHub repository. ## Conclusion So, these are the basic input techniques in Golang. We saw functions in the `fmt` package like `Scan`, `Scanf`, and `Scanln` which allow us to get input in a specific pattern. Hopefully, from this article part, we can build a firm base for further exploration like Strings, Arrays, and the ways to input them. Thank you for reading. If you have any questions or feedback, please let me know in the comments or on social handles. Happy Coding :) https://www.meetgor.com/posts/golang-conditionals-loops Introduction Moving to the fourth part, we will be doing conditional statements and loops in golang. We will be seeing the basics of conditional statements like Mon, 04 Apr 2022 00:00:00 UTC ## Introduction Moving to the fourth part, we will be doing conditional statements and loops in golang. We will be seeing the basics of conditional statements like if-else and switch along with loops like for, while, and range-based loops. We won't be covering iterating over arrays in a loop as this requires an understanding of arrays. ## Conditional statements Conditional statements are quite a fundamental aspect of learning a programming language. In golang, we have if-else conditional statements as well as switch cases. We will be exploring both of them in this section. Firstly, we will dive into if-else statements which are quite easy to understand. ### if else An `if` statement is used for checking the validity of a condition. If the condition is true(condition is met), a particular sets of statements are executed else (condition is not satisfied) different statements gets executed. We can use a basic `if-else` statement in go with the following syntax: ```go if condition { // statements }else{ //statements } ``` ```go package main import "fmt" func main() { age := 16 if age < 13{ fmt.Println("Kid") }else{ fmt.Println("Teenager") } } ``` ``` $ go run if_else.go Teenager ``` We can also use else if for evaluating more than one condition rather than using nested if and else. ```go if condition { // statements }else if condtion { //statements }else if condition { //statements }else { //statements } ``` ```go year := 2 if year < 1 { fmt.Println("Freshman") } else if year == 2 { fmt.Println("Sophomore") } else if year == 3 { fmt.Println("Junior") } else if year == 4 { fmt.Println("Senior") } else { fmt.Println("Probably a Graduate") } ``` ``` $ go run if_else.go Sophomore ``` Using `else if` we can evaluate multiple conditions. This style is much better than using nested `if else` statements as it becomes harder to read for complex cases. ### switch We also have switch statements in golang which allow us to write cases for a given state of a variable. We can simply add cases for a given variable, the case should be a valid value that the variable can take. If a case is matched it breaks out of the switch statement without executing any statements below the matched case. The basic structure of the switch statements in golang is as follows: ```go switch variable{ case value1: //statements case value2: //statements } ``` ```go age := 19 var result string switch { case age < 13: result = "Kid" case age < 20: result = "Teenager" case age < 25: result = "Adult" case age < 35: result = "Senior" } fmt.Println("The person is a",result) ``` ``` $ go run switch.go The person is a Senior with age 27. $ go run switch.go The person is a Teenager with age 19. $ go run switch.go The person is a Kid with age 11. ``` This gives a good understanding of switch-case statements. We can give a variable to the switch statement and pick its value in the respective case statements to evaluate the result accordingly. The `default` statement is evaluated when there is no match among the given cases. ```go language := "" var devs string switch language{ case "go": devs = "gopher" case "rust": devs = "rustacean" case "python": devs = "pythonista" case "java": devs = "Duke" default: language = "javascript" devs = "developer" } fmt.Println("A person who codes in",language,"is called a",devs) ``` ``` $ go run switch.go A person who codes in javascript is called a developer $ go run switch.go A person who codes in python is called a pythonista $ go run switch.go A person who codes in go is called a gopher ``` This code will by default pick `javascript` and `developer` as the values for `language` and `devs` respectively if there is no match for the provided language or the language is left empty. We also have `fallthrough` in the golang switch which allows evaluating more than one case if one of them is matched. This will allow the switch and check for all the cases sequentially and evaluate all the matched and satisfied cases. ```go character := 't' fmt.Printf("The input character is = %c ", character) switch { case character == 97: fmt.Printf("Its %c ", character) fallthrough case character < 107 && character > 96: fmt.Println("It's between a and k") fallthrough case character < 117 && character > 98: fmt.Println("It's between a and t") fallthrough case character < 122 && character > 98: fmt.Println("It's between u and z") default: fmt.Println("Its not a lowercase alphabet") } ``` ``` $ go run switch.go The input character is = f It's between a and k It's between a and t It's between a and u $ go run switch.go The input character is = k It's between a and t It's between a and u $ go run switch.go The input character is = a Its a It's between a and k It's between a and t It's between a and u ``` So, here we can see that the fallthrough hits multiple cases. This is unlike the base case which exits the switch statement once a case has been satisfied. This can be helpful for situations where you really want to evaluate multiple conditions for a given variable. ## Loops We can now move on to loops in golang. We only have a `for` loop so to speak but this can be used as any other looping statement like the `while` loop or range-based loop. We will first see the most fundamental loop statement in golang which is a three-component loop. ### for loop We can have a simple for loop in golang by using the three statements namely `initialize`, `condition`, and `increment`. The structure of the loop is quite similar to the other programming languages. ```go for k := 0; k < 4; k++ { fmt.Println(k) } ``` ### Range-based loop We can even iterate over a string, using the range keyword in golang. We need to have two variables for using a range-based for loop in golang one is the index or the 0 based position of the element and the copy of the element in the array or string. Using the range keyword, we can iterate over the string one by one. ```go name := "GOLANG" for i, s := range name{ fmt.Printf("%d -> %c ",i, s) } ``` ``` $ go run for.go 0 -> G 1 -> O 2 -> L 3 -> A 4 -> N 5 -> G ``` So, here we can see we have iterated over the string by each character. Using the range keyword in golang, The `i, s` is the index and the copy of the element at that index as discussed earlier. Using the index we get the value which we don't have to index the array for accessing it, that is already copied in the second variable while using the range loop. ### while loop (Go's while is for) There are no while loops as such in golang, but the for loop can also work similarly to the while loop. We can use a condition just after the for a keyword to make it act like a while loop. ```go for condition { // statements } ``` ```go count := 3 for count < 9 { fmt.Println(count) count++ } ``` ``` $ go run while.go 3 4 5 6 7 8 ``` We can see here that the condition is evaluated and the statements in the loop body are executed, if the condition evaluates to false, the flow is moved out of the loop and we exit the loop. ### Infinite loop We can run an infinite loop again using a keyword. We do not have any other keywords for loops in golang. ```go for { // statements // should have conditons to exit } ``` ```go flag := 4 for { flag++ fmt.Println(flag) } ``` This might be used with a base condition to exit the loop otherwise there should be a memory overflow and the program will exit with errors. ### Break If we want to exit out of a loop unconditionally, we can use the `break` keyword. This will break the loop and help us to exit out of an infinite or a condition-bound-based loop too. ```go flag := 1 for { fmt.Println(flag) flag++ if flag == 7 { fmt.Println("It's time to break at", flag) break } } ``` ``` $ go run infinite.go 1 2 3 4 5 6 It's time to break at 7 ``` As, we can see inside an infinite loop, we were able to break out of it by using a conditional statement and `break` keyword. This also applies to switch cases it basically is the opposite of `fallthrough` in switch-case statements. But by default(without using fallthrough), the case statement breaks the switch after a match has been found or the default case has been encountered. ### Continue We also have the opposite of `break` i.e. `continue` which halts the execution of the loop and directs back to the post statement increment(in case of for loops) or evaluation(in case of while loop). We basically are starting to iterate over the loop again after we encounter the continue but by preserving the counter/iterator state values. ```go counter := 2 for counter < 4 { counter++ if counter < 4 { continue } fmt.Println("Missed the Continue? at counter =", counter) } ``` ``` $ go run infinite.go Missed the Continue? at counter = 4 ``` For following up with the code for this and all parts of the series, head over to the [100 days of Golang](https://github.com/mr-destructive/100-days-of-golang) GitHub repository. ## Conclusion So, from this section, we were able to understand the basics of conditional statements and loops in golang. We covered the things which are more important for understanding the core of the language than some specific things. There are certain parts that need to be explored further like iterating over arrays and slices which we'll cover after we have understood arrays and slices. Hopefully, you have understood the basics of the conditional statements and loops in golang. Thank you for reading, if you have any questions, or feedback please let me know in the comments or social handles. Until next time, Happy Coding :) https://www.meetgor.com/posts/golang-variables Introduction In the third part of the series, we will be covering the fundamentals for learning any programming language i.e. variables and data types. We will Sun, 03 Apr 2022 00:00:00 UTC ## Introduction In the third part of the series, we will be covering the fundamentals for learning any programming language i.e. variables and data types. We will be covering from data types to variable declaration. We won't be seeing each and every detail related to the data types as some of them require a knowledge of loops and other topics, so that can be left for the different part. ## Types in golang In Golang there are 3 major types : Numeric, Bool and String. Further we also have specific types for the three data types like int, float, rune, byte, etc. We will first see how to declare a simple variable and then explore the data types in Golang. ```go var name string ``` This the variable declaration in Golang, we have the keyword `var` similar to Javascript but we optionally have to specify the type of the variable if we are not immediately assigning/defining it a value. To assign variable values, we can write the datatype of the assigned value but it is optional as the go compiler will know the datatype according to the assigned value. Further you cannot change the type of that variable once it is initialized. ```go var name string = "Gopher" OR var name string name = "Gopher" OR var name = "Gopher" ``` We also have `const` for assigning constant values to a particular value set. You cannot change the value for a constant type, doing this will result in compile time error. An important property of `const` can be noted here, if you simply declare a `const` without initializing the value and try to access that constant, the go-compiler will throw an compilation error. ```go const name string = "David" OR const name string name = "Calvin" OR const name = "Smith" ``` By default, the values for string is an empty string`""`, for integer and float it is `0` and for bool it is `false`. Each of these are valid declaration of variables in golang. Let's now dive into the data types and follow up with variable declaration in detail later. | Numeric | String | Bool | |------------|--------|------| | | | | | int | string | bool | | | | | | float | | | | | | | | complex | | | | | | | | rune | | | | | | | | byte | | | ### Numeric Let's first explore the `numeric` data types in golang as you have guessed correctly, we have `int` and `float` as distinct categories but further we also have fine grained storage types for both of the types. #### Integer In integers as well we have two categories `signed` and `unsigned`, we can basically store only positive integers in `unsigned` integers giving us an extra bit to play with. For Integers, we have specific data storage types depending on the bits it can store like `int8` for storing 8 bit integers, `int16` for storing 16 bit integer value, `int32` and `int64` for the given number of bits in the integer. Similarly we will have these storage based integer values for unsigned integers like `uint8`, `uint16`, `uint32` and `uint64`. We can basically store double amount of positive integers in unsigned integers as `uint` than in signed integers `int`, this is because the most significant bit is not used as a sign bit since all values in the variable are positive and hence no sign bit is required. ```go var likes int = 140 fmt.Println(likes) ``` ``` $ go run int.go 140 ``` ```go var age int8 = 140 fmt.Println("Age = ",age) ``` ``` $ go run int.go # command-line-arguments .\int.go:6:9: constant 140 overflows int8 ``` This will give us an error as `140` is above the limit for `int8`. So, unless you have specific requirements as storage limitation, you should be using `int` as the default data type for storing integers. So, we need to define variables as per the limits to which we are going to use them, if you just specify `int` the type will be selected based on your operating system, if it is `32bit`, it will take `int32`, for `64bit` OSes it will take as `int64` integer. If you define a variable with let say `16` bit storage and if it exceeds the limit for `16` bit storage, it would give a `overflow limit` error. Below are the limits for all the integer types in Golang: ``` uint8 -> 0 to 255 uint16 -> 0 to 65535 uint32 -> 0 to 4294967295 uint64 -> 0 to 18446744073709551615 int8 -> -128 to 127 int16 -> -32768 to 32767 int32 -> -2147483648 to 2147483647 int64 -> -9223372036854775808 to 9223372036854775807 ``` If you want to reality check for the boundary values of this data types, you can code a program in `go` as below: - To find the maximum value of uint, we can flip all the bits in `0` to get all the set bits in the integer thus we use `^` operator. - To find the maximum value for signed integers, we can right shit one bit so as to unset the sign bit. - The minimum value for uint is the default value `0`. - The minimum value for int can be calculated by subtracting one from the negative of the max limit. ```go package main import ( "fmt" ) func main() { var min_uint = 0 var max_uint8 uint8 = ^uint8(0) var max_uint16 uint16 = ^uint16(0) var max_uint32 uint32 = ^uint32(0) var max_uint64 uint64 = ^uint64(0) var max_int8 int8 = int8(max_uint8>>1) var min_int8 int8 = -max_int8 - 1 var max_int16 int16 = int16(max_uint16>>1) var min_int16 int16 = -max_int16 - 1 var max_int32 int32 = int32(max_uint32>>1) var min_int32 int32 = -max_int32 - 1 var max_int64 int64 = int64(max_uint64>>1) var min_int64 int64 = -max_int64 - 1 fmt.Println("uint8 -> ", min_uint, " to ", max_uint8) fmt.Println("uint16 -> ", min_uint, " to ", max_uint16) fmt.Println("uint32 -> ", min_uint, " to ", max_uint32) fmt.Println("uint64 -> ", min_uint, " to ", max_uint64) fmt.Println("") fmt.Println("int8 -> ", min_int8, " to ", max_int8) fmt.Println("int16 -> ", min_int16, " to ", max_int16) fmt.Println("int32 -> ", min_int32, " to ", max_int32) fmt.Println("int64 -> ", min_int64, " to ", max_int64) } ``` This was the basic overview of Integers in Golang. Though rune and byte are Integer aliases, we will explore them in the String section as they make a lot of sense over there. #### Float Similar to integers, we also have floats in the numeric category. A float is a numeric data type that can allow numbers with decimal values. A simple float can be of either `float32` or `float64`. The two types are defined as the precision of the decimal values. Obliviously, the `float64` type is more precise than the counterpart and is also the default type assigned if not provided. ```go const percent = 30.5 fmt.Println(percent+50) ``` You optionally provide the `float32` type to have a bit less precision than usual using the `float32` keyword as follows: ```go const percent float32 = 46.34 fmt.Println(percent - 3.555) ``` The floating value precision of the float types in golang are as follows: ``` float32 --> -3.4e+38 to 3.4e+38. float64 --> -1.7e+308 to +1.7e+308. ``` As quite logical reasons, the precision is almost double in the `float64` compared to `float32`. If we try to add(any operation) a `float64` number with a `flaot32`, we get an error as performing operations on two differently stored types can't be operated. #### Complex Numbers We also have complex numbers in golang. This are the numbers which deal with a real and a imaginary numbers. We initialize the complex variable using the `complex` function which takes two parameters the `real` part and the `imagianry` part. Both the parts or numbers are stored as float in the complex data type. So, since golang has `float32` and `float64` data types, we will have `complex64` and `complex128` as complex types. Since we are storing two `flaot64`, it has a `complex128` type and `complex64` for both parts as `float32`. Here as well, you cannot store the two parts(real and imaginary) as different float types i.e. You need to have both real and imaginary as either `flaot32` or `flaot64`. ```go var percent = 30.738 var f = 4.545 var comp1 = complex(f, percent) var comp2 = complex(percent, f) fmt.Println(comp1 - comp2) ``` ``` (-26.192999999999998+26.192999999999998i) ``` Golang automatically adds the `i` or iota in the complex/imaginary part for better readability. ### Strings We can now move onto the `string` data type in golang. It has several data types like `byte`, `rune`, `string`. In golang, `byte` and `rune` store individual characters whereas `string` can hold multiple characters. #### Byte A byte in golang is an unsigned 8 bit integer, which means it can hold numeric data from 0 to 255. So how is this displaying characters if it stores integer. Well, because each number it stores is mapped to the ASCII character set which is used to represent characters. A byte can be stored in a single quote `''`, if we use double quotes`""`, the variable is considered as string if we aren't specifying the data type. ```go const c byte = 't' fmt.Println(c) ``` ``` $ go run byte.go 116 ``` This gives the output as a number between 0 and 255 depending on the character which you have stored. To print the actual character you need to type caste into a string like: ```go const c byte = 't' fmt.Printf("Character = %c Integer value = %d ", c, c) ``` ``` $ go run byte.go Character = t Integer Value = 116 ``` We can get the character equivalent of the byte representation number using the [Printf](https://cs.opensource.google/go/go/+/go1.18:src/fmt/print.go;l=212) function and the `%c` place holder for a character. The ` ` is used to end the line just for displaying proper output. We can even store numbers between `0` and `255` as it is internally an `uint8`. #### Rune A rune is extended type of byte as it stores `int32` numbers and hence it represents `Unicode` characters. This is the default type if you do not specify `byte` and use single quotes to assign a character. Using rune, we can assign it an unicode characters to display some rich characters and literals like emoji or expressions. ```go var smiley_emoji = '☺' fmt.Printf("Smiley Emoji --> %c", smiley_emoji) ```  So, rune is pretty amazing type to play with characters in golang. As it is a default type assigned against byte if not provided while assignment. #### String Strings are basically a slice(list) of bytes. Each character in a string is a byte. By default the string will be empty if you don't initialize it with a value. ```go const language string language = "C++" OR const language string= "Python" OR const language = "Javascript" ``` We can even access particular character in the string using it's index. ```go const code = "12AB34CD" fmt.Println(code[6]) ``` ``` $ go run string.go 67 ``` This gives us a integer as we are accessing the byte from the string using its index. Thus, we can use `%c` in the `Printf` function to format and print the equivalent characters of the byte. ```go const code = "12AB34CD" fmt.Printf("2nd Character in string = %c ", code[4]) ``` ``` $ go run string.go 2nd Character in string = A ``` We can also declare strings using backticks/backquotes or whatever you call it (```), assigning string with this allows us to write multi line string. ```go var statement = `This is the first line The next line The last line` fmt.Println(statement) ``` ``` $ go run str-backticks.go This is the first line The next line The last line ``` Further in the loop article we will see how to loop/iterate over a string. ### Boolean This type is used to store either `true` or `false` in golang. The default value of a boolean variable is `false`. ```go var power bool fmt.Println(power) ``` ``` $ go run bool.go false ``` We can assign the variable as either `true` or `false`. ```go const result = true fmt.Printf("The statement is %t", result) ``` ``` $ go run bool.go The statement is true ``` So, using the `%t` we can print the value of a boolean value in golang in the `Printf` function. ## Creating Variables Now, we have familiar with data types in golang, we can more expressively create, declare, initialize variables in golang. There are 3-4 primary ways to define a variable most of which we have already seen. ### Declaring a Variable We can declare a variable without assigning it any value but for that we need to then provide the data type, this can be done using the following command: ```go var expereience int expereience = 2 ``` We can even use `const` for constant value in the given scope. Here, we can even declare multiple variables by separating each variable/constant with comma `,` which can be done as follows: ```go var a, b, c int OR const i, j, k bool ``` ### Defining and Initializing at the same time We can initialize a variable/constant in golang by explicitly giving it a value. We can do that by using `var` for variable value or `const` for a constant value. We can optionally provide the data type at this moment as golang will automatically detect the type and assign the memory according to the value given. ```go var place string = "home" ``` Here, there is no compulsion to provide the `datatype` as the compiler will be able to know it from the asisgned value. Though if you want to provide a non-default value you can specify the datatype. ### Declaring Multiple Variables We can assign multiple variables at once by separating them with comma`,`. The variable name to the left and the values to the right needs to separated with comm on both sides. ```go var x, y, z = 100, '#', "days" fmt.Printf(" x = %d y = %c z = %s ",x,y,z) ``` ``` $ go run multiplvar.go x = 100 y = # z = daysofcode ``` We can are declaring and assigning multiple variables, the `x` variable is assigned an integer value, `y` with a `rune`(by default) and `z` with a string. We are using `Printf` function with place holders for int `%d`, rune/byte `%c` and string as `%s`. The ` ` is for a new line. If we want to assign the variables with a particular data type, we can use the var keyword as a list of values. ```go var ( x int8 = 100 y byte = '#' z = "daysofcode" ) fmt.Printf(" x = %T y = %T z = %T ",x,y,z) ``` ``` $ go run multiplvar.go x = int8 y = uint8 z = string ``` This is not only limited to `var` we can also use `const` to declare multiple constants with type constraint. Also, note we are using the `%T` placeholder for getting the type of the data stored in the variable. Also, we can define(declare and initialize) multiple variable with same data type with command separated as follows: ```go var pi, e, G float32 = 3.141, 2.718, 6.67e-11 var start, end byte = 65, 90 fmt.Println(pi, e, G) fmt.Printf("%c %c ",start, end) ``` ``` $ go run multp.go 3.141 2.718 6.67e-11 A Z ``` ### Assigning Variable using Walrus Operator (Shorthand Declaration) We can skip usign `var` or the `datatype` by using the `:=` walrus operator. This type of assignment using `walruns` operator can only be allowed in the function body and not anywhere else, in the global scope this type of assignment is not allowed. ```go place := "school" ``` This is such a simple shorthand for assigning variables though only in a function body. Also, multiple variable declaration is possible with walrus operator. ```go x, y, z := "foo", 32, true fmt.Println(x, y, z) fmt.Printf("%T %T %T", x, y, z) ``` ```shell $ go run walrus.go foo 32 true string int bool ``` Links to all code and links are visible on the [GitHub](https://github.com/Mr-Destructive/100-days-of-golang) repository. ## Conclusion So, from this part of the series, we were able to understand variables and the various data types in Golang. Though we didn't got too much in detail still we can find ourselves a bit comfortable in understanding basic go scripts. In the next section, we will looking into conditional statements and loops. This would give a good grasp on iterating over a string and even learn arrays(just the basics) we will explore Arrays and slices(remember strings?) after that. So, if you have any questions, suggestions, or feedback please let me know in the comments or on the social handles. See you next time, Happy Coding :) https://www.meetgor.com/posts/hello-golang Introduction Moving on to the second day, we will be installing and setting up Go lang on our systems. The installation and setup are quite simple and not much Sat, 02 Apr 2022 00:00:00 UTC ## Introduction Moving on to the second day, we will be installing and setting up Go lang on our systems. The installation and setup are quite simple and not much demonstration is required, so further in the article, I will also make a hello-world program in GO. We will explore the basic program in GO and how to compile, run and build a GO program in this section. ## Installing Go Installing Go is pretty straightforward. You have to install the binaries from the official website as per your operating system. Head on to the [go.dev](https://go.dev/) which is the official website for the Go language. Click on the [Download](https://go.dev/doc/install) Tab and there should be all the required information. Install the installer as per your specific operating system. If you wish not to lead to any errors, keep the configuration for the installer as default and complete the installation process. ## Setting up Environment variables To make sure Go lang is perfectly installed in your system, type in CMD/Terminal/Powershell the following command: ``` go version ``` If you get a specific version of golang along with the platform and architecture of your system, you have successfully installed Go lang in your system. ``` $ go version go version go1.17.8 windows/amd64 ``` If you get an output as a command not found or anything else, this is an indication that your Go installation was not successful. You need to configure your Environment variables properly or re-install the installation script. ``` $ go version bash: go: command not found ``` ## Hello Gophers Once the Go lang has been successfully installed in your system, we can start writing our first program. Yes, a `Hello World` program. It is not as simple as `print("hello world")` but a lot better than 10-15 lines of Java or C/C++. ```go package main import ( "fmt" ) func main() { fmt.Println("Hello,Gophers!") } ``` So, this is so-called the `hello-world` program in Go, we will see each of the terms in detail next. But before that, let's get an idea of the style of the code. It is definitely similar if you are coming from C/C++ or Java, the package the main function. It will even feel like Python or Javascript when we explore other aspects. It has really a unique style of programming but feels familiar to programmers coming from any programming language, this might not be true for all programming languages though. ### Packages A package in Go lang is a way to bundle up some useful functions or any other constructs. Using packages we can reuse components of a specific app in another. Packages in golang also help in optimizing the execution/compilation time by letting the compiler only compile the required packages. Here, we have the main package, which provides the entry point for the entire project. This is mandatory for creating executable programs as we need to have a start point. The file name can be anything, but for executing the code, you need to have a main package where your main function resides. It provides an entry point as a package, when we will run the file, we provide the file which actually acts as a package and the file that has a tag of main is the entry point for the program. ### Main Function We have the main function where the main package is defined. It acts as a start point for a package. The main package will look for a main function inside the package. The main function doesn't take any parameter and nor does it return any value. When the function's scope ends, the program exits. The main function has significance only in the main package, if you define the main function in other packages excluding `main`, it works as a normal function. ### Import Statements We have an `import` keyword for importing packages from the standard library or other external packages from the internet. There are a lot of ways to import packages in golang like single, nested, multiple, aliased, dot import, and blank imports. We will see these different import styles in a dedicated section. Right now, we are directly importing a package, a single package. The pacakge is called the [fmt](https://pkg.go.dev/fmt) pacakge. It handles the format, input, and output format in the console. It is a standard package to perform some basic operations in golang. We can import it as a single direct import like: ```go import "fmt" ``` OR Make multiple imports like: ```go import ( "fmt" ) ``` We can add multiple packages on each line, this way we do not have to write the keyword `import` again and again. It depends on what you want to do in the program. ### Println function We can access the functions from the imported packages, in our case we can use the functions from the `fmt` package. We have access to one of the functions like `Println`, which prints string on a new line. Syntactically, we access the function and call it by using the `dot` operator like: ```go fmt.Println("Hi there!") ``` The [Println]() function takes in a parameter as a string and multiple optional parameters that can be strings or any variable. We will see how to declare variables and types in the next section. Here, the `P` in `Println` has significance as it allows us to distinguish private methods(functions in structs aka classes) from public methods. If a function begins with an uppercase letter, it is a public function. In technical terms, if the first letter of a method is upper case, it can be exported to other packages. ## Running Scripts Let's run the code and GO programming language to our resume. You can run a go source file assuming it has a main package with the main function using the following command: ``` go run <filename.go> ```  This will simply display the string which we have passed to the `Println` function. If you didn't have a main package this command won't run and return you an error: ``` package command-line-arguments is not the main package ``` By executing the run command, we can are creating a executable in a system's temp folder, For Windows, it's usually at: ``` C:\Users�cer\AppData\Local ``` You can get the location of the temp directory using CMD/PowerShell: ``` CMD: echo %TEMP% PowerShell: $env:Temp ``` For Linux ``` /tmp ``` You can get the location of the temp folder using Terminal in Linux/macOS: ``` echo $TMPDIR ``` It doesn't create any executable in the current project or folder, it only runs the executable that it has built in the temp folder. The run command in simple terms **compiles and executes the main package**. As the file provided to the run command needs to have the main package with the main function, it will thus compile that source code in the provided file. To get the location of the executable file that was generated by the `run` command, you can get the path using the following command: ``` go run --work <filename>.go ```  This will print the path to the executable that it currently has compiled. For further readings on the `run` command in Go, you can refer to the [documentation](https://pkg.go.dev/cmd/go#hdr-Compile_and_run_Go_program). ## Creating Executables We can go a step further by creating binary/executables with our source file using the `build` command: ``` go build <filename>.go ``` If you run this you would get an error as building a package requires a few things. The most important is the mod file. ``` go: cannot find main module, but found .git/config in D:\meet\Code\go\100-days-of-golang to create a module there, run: cd .. && go mod init ``` We need to create a mod file first before we build our script. A mod file in golang is the file that specifies the go version along with the packages and dependencies. It is like the `requirement.txt` but a bit different. We use the following command with the file that contains the main package among the other packages in the folder. We can even provide other packages to add to the mod file(see in detail in the future) ``` go mod init <filename>.go ```  This will generate a `go.mod` file, this is a file that contains the list of dependencies and packages in the project. If you look at the mod file, it looks as follows: ```go module script.go go 1.17 ``` Currently, this is pretty simple and has very little detail, but as your project increases in complexity, this file populates with the modules and packages imported and used in the project. So, after creating the mod file, we can build the script which we tried earlier. ``` go build <filename>.go ```  So, this command generates an exe right in the current folder. This will generate the file named after the package which is mainly `filename.exe`. If you have a `go.mod` file in your project, just running the command will generate the exe file: ``` go build ```  NOTE: For the above command to work, you need to be in the directory which has the mod file for your project. It basically bundles the listed packages and creates the executable named after the package which is named `main`. Thus it generates a different file name as `filename.go.exe` We can also provide an output file as the exe file name, this can be done with the following command: ``` go build -o <filename> ```  For further reading on the `go build` command, head over to this [documentation](https://pkg.go.dev/cmd/go#hdr-Compile_packages_and_dependencies) page. Link to all of the code and resources is mentioned in this [GitHub](https://github.com/Mr-Destructive/100-days-of-golang) repository. ## Conclusion So, from this second post, we were able to set up the Go language in our system and write our first `hello-world` program. This was a bit more than the setup and installation guide as it is quite boring to demonstrate the installation procedure being quite straightforward. Hopefully, you followed so far and you were able to pick up things in the go landscape. Thank you for reading and if you have any questions, suggestions, or feedback, let me know in the comments or the provided social handles. See you tomorrow, until then Happy Coding :) https://www.meetgor.com/posts/golang-intro Introduction This is the first entry in my 100 days of Go lang challenge. I will share the things I will learn each day for 100 days related to Golang or its co Fri, 01 Apr 2022 00:00:00 UTC ## Introduction This is the first entry in my 100 days of Go lang challenge. I will share the things I will learn each day for 100 days related to Golang or its configuration in my environment. In this post, we will understand what is golang and why you should learn it. It's not an in-depth guide but a high-level overview for understanding the context of learning Go lang. ### A bit of Context In April 2021, I started learning Django and to date, I have more than 10 projects(not all deployed) created and publicly open on [GitHub](https://github.com/Mr-Destructive?tab=repositories&q=django&type=&language=python&sort=). It has been a great experience so far, I am still in between an internship with a project to be created in Django. The learning process after I started blogging in June 2021 has been a different fantasy, I have been enjoying the process of writing the things I learn. So, this Spring/Summer I am creating a challenge of 100 days for learning Go lang and documenting the process. Hopefully, this will be a useful place to learn for a beginner and more than that a place for me to reflect back on the future. Leaving the rant aside, let's start with the best question to answer, What is Golang? ## What is Golang [Golang](https://go.dev/) (officially called Go) is a programming language developed by Google. It is statically typed like C\C++, Java, Kotlin, etc, and syntax is similar to dynamically languages like Python, Js, Ruby, Lua, etc. So, golang provides the best of both worlds, the swiftness to write code and also providing control and memory safety. Go is regarded as the language of the cloud i.e. it is heavily used in the cloud infrastructure, it involves creating microservices, APIs, containerization, orchestration, and a lot of other technologies. It can be a super useful programming language for DevOps folks and people into Cloud computing. Though Go is designed for Cloud Infrastructure, that doesn't mean you can't use it elsewhere. There are a lot of fields in which golang nicely fits. We'll discuss why you would want to learn golang and state the domains where it can be used. > Go programmers are called Gophers ## Why learn Golang? There are a lot of reasons for learning Go as your next programming language. Not only it is quite famous but it is significantly faster than other programming languages like python and javascript. It is a simple language to pick up, it is designed for modern world technologies. There is an added advantage to learning go lang as it is a new programming language(not as old as python). It was open-sourced in 2012, it got a great boost in the last five years as cloud companies are on the rise. ### 1. GO lang is simple GO lang is a simple programming language, it is really easy to learn. It is quite similar to python in terms of its syntax, though it also inherits characteristics from the Java/C++ paradigm. It has a single entry point program via the `main` function similar to Java/C++ languages and is compiled into binary/executable code before running. It has the standard library that provides a lot of functionalities out of the box. It provides some quite handy tools and has support for creating web applications in the core language itself. ### 2. GO lang is fast (Execution Speed) Though it might not be the fastest language of all time, it beats python and javascript by a significant magnitude in the context of backend development. Go lang is compiled language and hence it has no overhead of converting into byte code or the interpreter. Go lang also checks for unused variables, functions, identifiers, and packages, this can boost the speed of execution by a good amount additionally optimizing the memory footprint of the program.  We can see, that Go lang is almost equivalent to C and C++ porgramming language, though this might not be the best comparison of speed as it depends on quite a lot of factors and the context of programs. Still Go lang has a significant edge over Java and C++ which is a great factor because of the writing complexity involved in those programming languages. ### 3. GO lang is fast (Writing Speed) GO language is fast yes I get that, It is faster in terms of writing speed. If you sit down to write the same code in Java/C++ or Go, having time constraints, you will move out with Go as a winner. It is much easier to write code in Go than compared to C/C++, Java, or even Javascript. Golang prefers clear code over fast code, though it might not be as fastest as Rust or C++, it is definitely much easier and quicker to write than the other counterparts. Go is designed for writing software that can scale with large codebases and teams. So it's not a surprise it is used to create cloud computing technologies like Docker, Kubernetes, and other tons of cloud-native tools. ### 4. GO lang is made for the modern world > Go lang is the modern C Go lang is a modern language, it is not as ancient as C/C++ or Java. It is designed for Cloud-native applications which are scalable and optimized for performance meeting the needs of the modern demand for bleeding-edge technologies. It is designed by the people who created the UNIX operation, Ken Thompson. Though there were other go authors like Robert Griesemer and Rob Pike. Go lang has been designed keeping the idea of C but also considering the context of the modern world. It makes tasks like concurrency a lot simpler but it avoids dangerous and manual control of memory by not allowing pointer arithmetic. ## Where you can use Golang Go land finds its use case in a lot of domains including Cloud computing, Web applications, CLI and DevOps tools, and so on. As GO is a general-purpose programming language, you can use it anywhere in the relevant fields of technology. ### Cloud-native application As Golang has capabilities to execute low-level system calls efficiently, cloud-native tools can be created in golang for improving efficiency and memory usage. As Go uses fewer resources like CPU and memory, it can optimally be used for applications that require almost no delay in the response. It also handles concurrency very well using goroutines which are way simpler to implement. All the factors contribute very well to constructing a modern cloud-native application. Golang is used in Cloud-native technologies like - [Docker](https://github.com/docker) - containerisation technology - [Kubernetes](https://github.com/kubernetes/kubernetes) - Container Orchestration - [Google Cloud](https://github.com/google/go-cloud) - Cloud Hosting Plaform This makes Golang widely adopted and has firm grounds for the future as well. ### Web applications As Golang is highly efficient and fast it can be used to develop web applications. It can scale a web application very well as it is a statically typed ecosystem. It also has extensive support for integrating databases into web applications. Golang doesn't require any 3rd party integrations to create the web apps, it is developed using internal or external packages and modules. The performance of Golang is the core feature for creating web apps in it, it heavily outperforms Javascript(Node) and Python. As golang code is compiled before running, this makes a single executable depending on the operating system, this improves the performance to a great extent. So, golang should be an ideal choice for developing web applications. - [Soundcloud](https://developers.soundcloud.com/blog/go-at-soundcloud) - Music System Webapp - [Uber](https://www.slideshare.net/RobSkillington/go-at-uber) - Cab Booking/Ride Sharing App - [Allergo](https://github.com/allegro/marathon-consul/#marathon-consul-) - E-commerce Webappp ### DevOps tools A lot of DevOps tools like Prometheus, Grafana, Terraform, Hugo are written natively in Go. Having appropriate features like simplicity, scalability, efficiency, and concurrency, go lang fits the DevOps space really well. Golang is already used in cloud-native technologies as mentioned earlier adopting it in the DevOps space is a great choice as it creates an even and supportive ecosystem. The `Write once and Run anywhere` paradigm of java is surprisingly built-in golang as well without conversion into byte code. This is a great feature any DevOps engineer would dream of, using a single code base for multiple platforms can save a lot of time and improve productivity. - [Prometheus](https://github.com/prometheus/prometheus) - Monitoring system and time series database - [Grafana](https://github.com/grafana/grafana) - Platform for monitoring and observability - [Terraform](https://github.com/hashicorp/terraform) - Building, Changing, and Versioning Infrastructure - [Minikube](https://github.com/kubernetes/minikube) - Run kubernetes locally - [Kit](https://github.com/go-kit/kit)- standard library for microservices ## Who uses Golang? Golang is a widely used programmming language, it is used by some of the big companies around the world. Of course Google uses it, apart from Google, it is being used by companies like: 1. Netflix 2. PayPal 3. Microsoft 4. Dropbox 5. Meta (facebook) 6. Uber 7. Twitter 8. Twitch There are so many tools created tools using golang like the [gh-cli](https://github.com/cli/cli), [cockrach-db](https://github.com/cockroachdb/cockroach), [jwt](https://github.com/dgrijalva/jwt-go), etc. You can see a whole list of applications/tools/frameworks written in Go from this [awesome list](https://github.com/avelino/awesome-go) ## Conclusion Go lang is a excellent choice for any one into web developement, cloud native, DevOps, CLI tools. Go lang being a recently developed language it has grown a lot faster than any other language in the programming world. Learning Python over the span of two years, I might want to dabble into Go but not completely leave Python. Python has a great community and is widely adopted in almost all fields in developement. Still trying out new things is what keeps you in the hunt of oppurtunitiees and meeting new people. So starting from today, I will share my learning a new thing in Go lang every day for 100 days. At least that's the plan. The post might be in the form of a long post or a short TIL on the blog. Check out the GitHub repository for the 100 day challenge [here](https://github.com/mr-destructive/100-days-of-golang).If you have any queries, suggestions, or feedback please let me know in the comments or on the mentioned social handles. Thank you for reading. Happy Coding :) https://www.meetgor.com/posts/python-feedparser Introduction is a simple but powerful python package that can be used to extract information about a specific webpage or a publication with its RSS feed(not onl Sat, 26 Mar 2022 00:00:00 UTC ## Introduction [Feedparser](https://pypi.org/project/feedparser/) is a simple but powerful python package that can be used to extract information about a specific webpage or a publication with its RSS feed(not only RSS). By providing the RSS feed link, we can get structured information in the form of python lists and dictionaries. It can be basically used in a pythonic way to read RSS feeds, it is really simple to use and it even normalizes different types of feeds. Today, we will be taking a look at the feedparser package in python and how to extract information from a given RSS feed. ## What is feedparser Feedparser is a python package for parsing feeds of almost any type such as RSS, Atom, RDF, etc. It is a package that allows us to parse or extract information using python semantics. For example, all the latest posts from a given blog can be accessed on a list in python, further different attributes like links, images, titles, descriptions, can be accessed within a dictionary as key-value pairs. ## Installing feedparser As feedparser is a python package you can install it with pip very easily. ``` pip install feedparser ``` This will install feedparser in your respective python environment, it can be a virtual environment or a global environment. ## Using feedparser To test out feedparser, you can open up a python repl, in the environment where you installed the Feedparser package. ``` python ``` Firstly import the package. ```python import feedparser ``` Now, we can use the module in our application to get all of the functions or methods from the package. ## Parse an RSS feed URL To parse an RSS feed link, we can simply use the `parse` function from the feedparser package. The [parse](https://feedparser.readthedocs.io/en/latest/introduction.html) function takes in a string that can be a URL or a file path. Generally, the URL seems to be more useful. So, we can look up any RSS feed on the internet like your blog's feed, publications feeds, and so on. ```python feedparser.parse("url_of_the_rss_feed") ``` The parse function basically fetches the feed from the provided URL or the file. It extracts the feed in a systematic way storing each piece of information in a structured format. At the high level, it returns a dictionary with a few key-value pairs. Further, each key might have a list or nested dictionaries in it. The key identifiers are named in a uniform manner for any feed you parse in the function. Though there might be a few cases where there might be additional information to be parsed, it can even add more information ad shape the structure accordingly. This will give you a dictionary in python, that can have more or less similar keys. The most common keys that can be used in extracting information are `entries` and `feed`. We can get all the keys associated with a feed that is parsed using the `keys` function. ```python feedparser.parse("url_of_the_rss_feed").keys() ```  The keys function basically gets all the keys in the dictionary in python. ``` >>> feedparser.parse("https://dev.to/feed/").keys() dict_keys(['bozo', 'entries', 'feed', 'headers', 'etag', 'href', 'status', 'encoding', 'version', 'namespaces']) ``` This will give out a list of all the keys in the feed which we have parsed from the RSS feed previously. From this list of keys, we can extract the required information from the feed. Before we extract content from the feed, we can store the dictionary that we get from calling the parse function. We can assign it to a variable and store the dictionary for later use. ```python feed = feedparser.parse("url_of_the_rss_feed") ``` ## Extract the contents from the feed Now, we have the dictionary of the feed, we can easily access the values from the listed keys. We can get the list of all the posts/podcasts/entries or any other form of content the feed is serving for from the `entries` key in the dictionary. To get more information and the most possible keys in the returned dictionary, you can refer to the feedparser [reference list](https://feedparser.readthedocs.io/en/latest/reference.html) ### Access Articles from Feed To access the articles from the feed, we can access those as a list of the articles. Using the `entries` key in the dictonary as follows: ```python feedparser.parse("url_of_the_rss_feed")["entries"] OR feedparser.parse("url_of_the_rss_feed").entries ``` If you have already defined a variable set to the parse function, you can use that for more efficient extraction. ```python feed = feedparser.parse("url_of_the_rss_feed") feed['entries'] OR feed.entries ``` ### Get Number of Articles/Entries from Feed To get the number of entries in the list, we can simply use the len function in python. ```python len(feed.entries) OR len(feedparser.parse("url_of_the_rss_feed").entries) ```  This gives us the number of entries in the provided feed. This is basically the list that stores all the content from the publication/website. So, we can iterate over the list and find all the different attributes we can extract. ### Get details of the entries from the feed To get detail information about a particular article/entry in the feed, we can iterate over the `feed.entries` list and access what we require. So, we will iterate over the entries and simply print those entries one by one to inspect what and how we can extract them. ```python for entry in feed.entries: print(entry) ``` It turns out that every entry in the list is a dictionary again containing a few key-value pairs like `title`, `summary`, `link`, etc. To get a clear idea of those keys we can again use the keys function in python. ```python feed = feedparser.parse("url_of_the_rss_feed") print(feed.entries[0].keys()) ```  ```python >>> feed.entries[0].keys() dict_keys(['title', 'title_detail', 'authors', 'author', 'author_detail', 'published', 'published_parsed', 'links', 'link', 'id', 'guidislink', 'summary', 'summary_detail', 'tags']) ``` Now, we have all the keys associated with the entries we can now extract the specific details like the content, like `title`, `author`, `summary_detail`(actual content in this case). Though this might not be the same for all RSS feeds, it might be very similar and a matter of using the right keyword for the associated keys in the list of dictionaries. Let's say, we want to print out the titles of all the entries in the feed, we can do that by iterating over the entries list and fetching the title from the iterator as `entry.title` if `entry` is the iterator. ```python for entry in feed.entries: print(entry.title) ```  Similarly, we will get the links of the entries using the link key in the dictionary. ```python for entry in feed.entries: print(entry.link) ``` ### Get information about the Website/Publication To get the metadata about the information you are extracting from i.e. the website or any publication, we can use the key `feed`. This key stores another dictionary as its value which might have information like `title`, `description` or `subtitle`, `canonical_url`, or any other data related to the website company. ```python feed.feed or feedparser.parse("url_of_the_rss_feed").feed ```  From this dictionary, we can now simply extract the specific information from the keys. But first, as in the previous examples, we need a clear idea of what are the keys in the dictionary where we can extract the specific value. ```python feed.feed.keys() or feedparser.parse("url_of_the_rss_feed").feed.keys() ``` Using the keys like `title`, `links`, `subtitle`, we can get the information on the website/company level and not related to the specific post in the entries list. ```python # get the title of the webpage/publication feed.feed.title # get the links associated with the webpage feed.feed.links # get the cover-image for the webpage feed.feed.image ``` You can further get information specific to your feed. ## Checking for keys existence in the dictionary of feed We also need to check for the existence of a key in a dictionary in the provided feed, this can be a good problem if we are parsing multiple RSS feeds which might have a different structure. This problem occurred to me in the making of [podevcast](https://podevcast.netlify.app) where I had to parse multiple RSS feeds from different RSS generators. Some of the feeds had the cover image but most of them didn't. So, we need to make sure we have a check over those missing keys. ```python feedlist = ['https://freecodecamp.libsyn.com/rss', 'https://feeds.devpods.dev/devdiscuss_podcast.xml'] for feed in feedlist: feed = feedparser.parse(feed) print(feed.entries[0].keys()) for entry in feed.entries: if 'image' in entry: image_url = entry.image else: image_url = feed.feed.image #print(image_url) ``` ```python >>> feedlist = ['https://freecodecamp.libsyn.com/rss', 'https://feeds.devpods.dev/devdiscuss_podcast.xml'] >>> for feed in feedlist: ... feed = feedparser.parse(feed) ... for entry in feed.entries: ... if 'image' in entry: ... image_url = entry.image ... else: ... image_url = feed.feed.image ... print(feed.entries[0].keys()) ... dict_keys(['title', 'title_detail', 'itunes_title', 'published', 'published_parsed', 'id', 'guidislink', 'links', 'link', 'image', 'summary', 'summary_detail', 'content', 'itunes_duration', 'itunes_explicit', 'subtitle', 'subtitle_detail', 'itunes_episode', 'itunes_episodetype', 'authors', 'author', 'author_detail']) dict_keys(['title', 'title_detail', 'links', 'link', 'published', 'published_parsed', 'id', 'guidislink', 'tags', 'summary', 'summary_detail', 'content', 'subtitle', 'subtitle_detail', 'authors', 'author', 'author_detail', 'itunes_explicit', 'itunes_duration']) ``` As we can see we do not have an image key in the second RSS feed which means each entry doesn't have a unique cover image, so we have to fetch the image from the `feed` key then the `image` key in the entries list.  As we can see here, the image_url will pick up the `image` key in the dictionary if it is present else we will assign it to another URL which is the website/podcast cover image. This is how we handle exceptions in providing the keys when there are multiple feeds to be extracted though they are quite similar, they will have subtle changes like this that need to be handled and taken care of. ## Conclusion From this little article, we were able to understand and use the feedparser Python package which can be used to extract information from different feeds. We saw how to extract contents for the entries, a number of entries in the feed, check for keys in the dictionary, and so on. Using Python's Feedparser package, some of the projects I have created include: - [podevcast](https://podevcast.netlify.app) - [dailydotdev-bookmark-cli](https://pypi.org/project/dailydotdev-bookmark-cli/) - [Django Newsletter](https://github.com/Mr-Destructive/newsletter) For further reading, you can specifically target a feed type by getting the appropriate methods from the feedparser [documentation](https://feedparser.readthedocs.io/en/latest/) Thank you for reading, if you have any suggestions, additions, feedback, please let me know in the comments or my social handles below. Hope you enjoyed reading. Happy Coding :) https://www.meetgor.com/posts/pipx-intro Introduction Previously, I explored the as a python package management tool and this week it is time for exploring a python CLI package isolation tool called . Sun, 20 Mar 2022 00:00:00 UTC ## Introduction Previously, I explored the [pipenv](https://mr-destructive.github.io/techstructive-blog/pipenv-intro/) as a python package management tool and this week it is time for exploring a python CLI package isolation tool called `pipx`. It is a really simple and powerful tool for running commands as an executable from an isolated environment. Those commands should be associated with a python package that has CLI. In pipx, we install the package once and we can use the package anywhere on our system isolated from other virtual environments. In this article, We will explore the pipx package/tool in python. From installation to setting up your environment and removing some friction in your python workflow. This won't be an in-depth guide of pipx, though we would cover a few python packages that can be easily used in your environment. ## What is Pipx Pipx is a python package that works similar to `pip` but it is quite specific to the usage. It can run CLI commands of any python package from anywhere on your system. It uses its own virtual environment for managing the packages and the python version. **NOTE** - Pipx is a tool to install and execute CLI commands provided in the python packages - Pipx is not a python package management tool like pip or pipenv. Pipx is similar to `npx` for nodejs. Though the major difference is that pipx runs and installs all packages/tools globally unlike npx. ## Install Pipx **Pipx requires Python 3.6 or above.** - You can install it normally like a python package with pip: ``` pip install pipx ``` To find out if pipx was successfully installed, you can simply type in the command: ``` pipx --version ``` If it gives a number and no errors, it has been successfully installed in your system globally. ## Pipx Commands The primary commands that pipx can use are: 1. pipx install 2. pipx list 3. pipx run 4. pipx inject 5. pipx uninstall For further command options for a specific use case, you can use the `pipx -h` command to get the detailed list of commands and options available in pipx. Also, the documentation of pipx is really incredibly beginner-friendly. You can follow up with the article from the [documentation](https://pypa.github.io/pipx/) webpage or the [GitHub](https://github.com/pypa/pipx/) repository. The documentation and the webpage are simple single-page websites that really make learning the tool easier and it also has a well-documented package. ### Pipx Install Command Pipx can install packages in a virtual environment specific to the package provided. This means that you will have different virtual envs for each package you install and not a single virtual env for a project, it creates and makes the package venvs accessible globally. The next step after installing the package in a different venv is to make that package commands accessible via the PATH environment variable. ``` pipx install <package_name> ``` If say we install [pgcli](https://pypi.org/project/pgcli/), a package for interacting with postgres database using python in the Command line. We will simply enter the following commands: ``` pipx install pgcli ```  We can use the package pgcli globally in our system. So, we have access to the databases in the local Postgres server. We can run the commands as normally as we want using the pgcli by just prefixing with `pipx run`. ``` pipx run pgcli <local_database_name> -U postgres ```  This is a really cool way to interact with CLI applications without installing separately or globally, using pipx the virtual environment for the specific package makes it a clean and persistent behavior for running the commands anywhere in the system. If you want to know more about pgcli tool/package in Python, you can it's [documentation](https://www.pgcli.com/docs) webpage or the [GitHub](https://github.com/dbcli/pgcli) repository. #### Pipx Default PATH The default path at which pipx stores the virtual environments for each package is by default the `~/.local/pipx/venvs`(here ~ refers to the root/default directory), the documentation says that it can be changed by editing the environment variable `PIPX_HOME`. The default path at which pipx stores the binaries/command apps for each package is stored in the path `~/.local/bin`, even this path can be modified by editing the environment variable `PIPX_BIN_DIR` ### Pipx List Command Pipx list command simply lists the executables or the commands you can run with pipx. This command will display the commands associated with the packages that are installed in the pipx environment. ``` pipx list ```  This command gives a detailed list of the commands associated with their respective packages. It also gives the meta-information like the Python version in which the package was installed. ### Pipx Run Command The most useful command in pipx has to be `pipx run`, this command can execute provide package along with the specified command associated with the package. The structure of the command is as follows: ``` pipx run <package_name> <command> ``` The pipx documentation refers to the `command` as an `APP`, as it is an executable created from the package specification. If the APP/command name is not matching the `package_name`(most of the time it won't), you need to add an argument to the run command. ``` pipx run --spec <package_name> <command> ``` The `--spec` option allows specifying a certain package to be used while running the command/APP. Using this option with the `pipx run` command, we can run package-specific commands. For example, if we want to run an ipython from the terminal, without messing up the current environment i.e. without installing any package. You can do that by using pipx. ``` pipx install dailydotdev-bookmark-cli pipx run --spec dailydotdev-bookmark-cli bookamrks ```  This is my shameless plug of installing my [first python package](https://github.com/Mr-Destructive/bookmarks-cli). The command used for the CLI is totally not related to the package name and hence we have to use the `--spec` option. ## Install and Run packages from Pipx Any python package which provide a Command Line Interface can be installed and run with pipx. Some of the packages like [pytest](https://docs.pytest.org/en/7.1.x/contents.html), [pipenv](https://pipenv-fork.readthedocs.io/en/latest/install.html), [rich-cli](https://github.com/Textualize/rich-cli), [Black code formatter](https://github.com/psf/black), [markata](https://markata.dev/) and many others. You can install any of the packages which do have a cli to interact with on the go with pipx. ### Running Ipython shell We can use [Ipython](https://pypi.org/project/ipython/) shell from pipx as an isolated environment. We simply install the package first, after installing the package it creates the virtual environment. Creating a separate virtual environment registers the app/command binaries into the PATH environment of the system so that they can be accessed globally. ``` pipx install ipython pipx run ipython ```  For further documentation on ipython using the CLI, you can refer to the [GitHub](https://github.com/ipython/ipython) link or the [documentation](https://ipython.org/ipython-doc/3/interactive/reference.html#command-line-usage) page. ### Reading an IPYNB file from pipx with JUT If you just want to read an ipynb file from the terminal, you can do that using [jut](https://pypi.org/project/jut/) by specifying the command: ``` pipx install jut pipx run jut <notebook.ipynb> ```  You can find more information on JUT via their official [GitHub](https://github.com/kracekumar/jut) repository. ### Using rich-cli to display text using pipx We can even use [Rich-CLI](https://pypi.org/project/rich-cli/) to print rich content in the project. Rich-CLI as the name suggests is a CLI for the Rich package in python. It is really intuitive and simple to use. If we want to take snaps of the rich content of a markdown file, source file, dataset, etc. rich-cli is a tool that can quickly do that. Use pipx to install the package globally and simply run wherever required in an isolated environment. ``` pipx install rich-cli pipx run --spec rich-cli rich "[blue]Hello[/], [yellow]World[/]!" --print ```  This will give you a colorful display of "Hello, World" without you messing up your current project. Pipx has an isolated environment so it doesn't tinker with your local virtual environment or project. Though you can use your current project to use those CLI to execute commands which might require the source file in the project. We can even print the markdown file in a rich format, simply using pipx and rich-cli. There are a lot of things we can do with rich-cli. ``` pipx run rich-cli <markdown_file.md> ```  For further reading on the rich-cli package, you visit their [GitHub](https://github.com/Textualize/rich-cli) link and the [Documentation](https://github.com/Textualize/rich-cli/blob/main/README.md) Readme. ### Using pytest to perform a test for the current project [Pytest](https://pypi.org/project/pytest/) allows writing simple and scalable tests for Python apps, libraries, and packages. We can use it to write tests in the project without adding it as a dependency in the python environment. Simply install the package with pipx and run the tests in an isolated environment. ``` pipx install pytest pipx run pytest temp.py ```  Here, we can see the pytest was performed on the `temp.py` file which can be any application file for your project. This gave us results without installing pytest in our application's virtual environment. This becomes really convenient to run certain package commands whose package should not be a dependency on your project in the virtual environment. You can visit Pytest's [GitHub](https://github.com/pytest-dev/pytest/), [Documentation](https://docs.pytest.org/en/stable/) and their [Home Page](https://pytest.org/). ### Using Black (Python code formatter) We can use pipx to even format the python source files using [black](https://pypi.org/project/black/). We don't have to install `black` as a dependency in the project. Pipx will simply install black in its isolated environment and run the specified command. ``` pipx install black pipx run black temp.py ```  Black is a code formatter in Python, it basically edits your source python files for any incorrect python semantics and syntax, it corrects/formats them without a compromise. You can visit the [GitHub](https://github.com/psf/black), [Documentation](https://black.readthedocs.io/en/stable) or the Black package [Playground webpage](https://black.vercel.app/). ### Using httpie (curl equivalent in Python) We can use the [httpie](https://pypi.org/project/httpie/) package in python which is a CLI tool for HTTP clients. It is very similar to the `curl` command in Linux. We can even use that with pipx to test out API endpoints or any website that you might be working with without leaving the terminal. For further details on the httpie package, you can visit their [Home page](https://httpie.io/), [GitHub](https://github.com/httpie), or [PyPI webpage](https://pypi.org/project/httpie/). ``` pipx install httpie pipx run --spec httpie http -v dev.to ```  Here, we can see the package simply gives a simple output of the headers of the request. It is smart enough to understand the web link as `https://dev.to` unlike CURL which needs to have an exact match. ### Running Markata blog with pipx As said, every package that bundles itself with a CLI can be installed and run from anywhere. So, [markata](https://pypi.org/project/markata/) which is a plugin-driven static site generator which is a python package also bundles with a CLI. Using the base app(markata) we can run its associated commands like `build`, `list`, etc. I use this to build my blog. ``` pipx install markata pipx run markata build ```  You can visit the Markata's [documentation](https://markata.dev/) web page or the [GitHub](https://github.com/WaylonWalker/markata) repository for further insights on the Static site generator. **If you want to explore more packages that you can use with pipx, then you can look at the docs guide [programs-to-try](https://pypa.github.io/pipx/programs-to-try/) section.** ## Conclusion So, from this simple introduction, we were able to understand the basics of pipx which is a python package for isolating and running CLI-specific package commands. We saw how to setup pipx, install packages, run the commands from the package and interact with the virtual environment that it creates for each package we install. Thank you for reading, if you have any comments, suggestions, feedback please let me know in the comments. Happy Coding :) https://www.meetgor.com/posts/pipenv-intro Introduction Pipenv is Python's officially recommended Package management tool, as the name suggests it combines the power of and . It is really powerful and gi Mon, 14 Mar 2022 00:00:00 UTC ## Introduction Pipenv is Python's officially recommended Package management tool, as the name suggests it combines the power of `pip` and `virtualenv`. It is really powerful and gives control of package management for a python project. It also has support for adding dev packages, python versions, smartly recognizing main and derived package installation/uninstallation, and so on. It is the [official package management tool for Python](https://packaging.python.org/en/latest/tutorials/managing-dependencies/#managing-dependencies). It is quite similar to npm for Nodejs in Javascript, or bundle for Ruby, cargo for Rust, and so on. It really simple and easy to use as it manages the overhead for package management for us and hence it is also a high-level package management tool as opposed to pip, which is not as powerful as Pipenv. So, in this article, we'll explore [Pipenv](https://pypi.org/project/pipenv/) package manager for Python and how you can use it in your next python project. Let's get started. ## Install Pipenv [Pipenv](https://pypi.org/project/pipenv/) is just like any other package in python, you can install it with pip as normally you install any other package with the command: ``` pip install pipenv ```  You can refer to the documentation of Pipenv from [here](https://pipenv.pypa.io/en/latest/). ## Set up a Python Environment This step is not mandatory but it avoids any mistakes that you can make in the future while installing the package, so to simply create a new python environment for your project, you simply have to write the following command: ``` pipenv shell ```  This will create the virtual environment for Python in the local folder. It is quite similar to installing `virtualenv` and then activating the env/venv folder script. Though pipenv is quite powerful as it automatically detects the environment. If you look at it carefully, there will be a file generated after the command has been executed successfully. The file called `Pipfile` without any extension will have been created in your current folder where you executed the command from. The file contains the version of python used in this project along with the list of dependencies(currently empty). Also the source from where it will download and manage the dependencies. The Pipfile after simply creating the virtualenv via the command `pipenv shell` looks something like follows: ``` [[source]] url = "https://pypi.org/simple" verify_ssl = true name = "pypi" [packages] [dev-packages] [requires] python_version = "3.8" ``` If you want to set up a specific version of python for the virtual environment, you can do this using pipenv. The version should be installed in your system though for the pipenv to pick it up, if you have the specific version of python set up with all path variables configured, you can use the below command to set up the version of python in pipenv. ``` pipenv --python version-number ``` For example : `pipenv --python 3.9` will set up the virtual environment with python version `3.9`. ## What is Pipfile Pipfile is basically a TOML file that has all the details of the different dependencies/packages and the version of Python used in the project/directory. A TOML is a simple configuration file that is reader-friendly, it is a map of keys and values as configuration data structures. In Pipenv, we can have keys like `package-names` and the value as the `version-number`, certain groups of dependencies like `dev-packages` and `packages`, etc. Pipenv is the file that Pipenv implements its package management environment. The file is really important and powerful as it can install all dependencies even if provided with `requirements.txt` file. Yes, it can automatically detect that if you provide the path to that file. Pipenv also has additional features like adding dev dependencies/packages in the project as a separate dependency. So if you want to test a feature with a specific package you can add it as a dev package and it will be stored separately. The pipfile will segregate the dependencies so that Pipenv can install/uninstall from the virtual environment. In short, Pipfile lets you have great control over your project's packages management. ## Installing Python Packages Once your Pipenv is initialized as a virtual environment for Python, we can install dependencies with either `pip` or `pipenv`. This is the mistake that might get you trapped, if you already have not run the command `pipenv shell` and installed any dependencies with `pip`, you will install that dependency globally and make some mess of the project. So, it is advised to use `pipenv shell` in order to activate your virtual environment. If you do not wanna do that, you can use the command to install any dependency without activating the virtual environment. ``` pipenv install <package-name> ```  This will detect any virtual environment in the project, if it's not created already it will create it and install that package in that environment. Installing any package using `pipenv` will update the Pipenv file and the package to its packages list. ```toml [[source]] url = "https://pypi.org/simple" verify_ssl = true name = "pypi" [packages] flask = "*" [dev-packages] [requires] python_version = "3.8" ``` OR If you wish to install with pip, as usual, you need to be in the virtual subshell. If you already are, then Pipenv will add that dependency to the virtual environment. Note though, if you install any package with `pip` and not with `pipenv`, the package won't be added to Pipfile but would be installed in the virtual environment.  Here, if we type the command `pipenv graph` it will show us a nice detailed list of all the installed dependencies. ``` pipenv graph ``` ```shell $ pipenv graph Flask==2.0.3 - click [required: >=7.1.2, installed: 8.0.4] - colorama [required: Any, installed: 0.4.4] - itsdangerous [required: >=2.0, installed: 2.1.1] - Jinja2 [required: >=3.0, installed: 3.0.3] - MarkupSafe [required: >=2.0, installed: 2.1.0] - Werkzeug [required: >=2.0, installed: 2.0.3] requests==2.27.1 - certifi [required: >=2017.4.17, installed: 2021.10.8] - charset-normalizer [required: ~=2.0.0, installed: 2.0.12] - idna [required: >=2.5,<4, installed: 3.3] - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.8] ``` As you might have guessed, the above command is equivalent to the `pip freeze` command, but just compare the details both tools have. Pipenv really shines here. If you compare the output of Pipfile and `pipenv graph` you get a bit confused as to why is there are so fewer packages in the file. So, **Pipfile doesn't store the sub-packages/dependencies of a certain base package**. Let's take, for example, Flask here, we have Flask as the main package, and `click`, `Jinja2`, `Werkzeug` are its sub dependencies, further `colorama` and `MarkupSafe` are in turn dependencies of `click` and `Jinja2`. So, Pipfile just includes the top-level packages, pipenv is smart enough to install the rest. It also doesn't include `requests` but it is indeed included in the `graph`. This is because Pipenv will only put the dependencies in the Pipfile if it has been installed via the `pipenv` command.  You can use `pip freeze` here as well as we are technically in a Python virtual environment. And you can clearly see the difference. Pipenv is a high-level tool compared to pip, it technically uses `pip` and `virtualenv` and leverages both of them to become a smart package management tool. ## What is the Pipfile.lock - If you are coming from `Javascript` world, it is similar to `package-lock.json` file. - If you are coming from `Ruby` world, it is similar to the `Gemfile.lock` file. - If you are coming from `Rust` world, it is similar to the `cargo.lock` file. Ok, you get the idea, it is a file that more robustly specifies the version of the packages without conflicting with the other version or the Python version itself. If you look at the Pipfile.lock also has hashes that store the sub-packages as well. The file format here is JSON as opposed to TOML for the Pipfile. ## Configuring the Pipenv environment Now, a question you might have is where is the virtual environment? Is it there? Of course, it will be there, it is configured to a different location though. By default, it will be stored in the `~\.virtualenvs\` folder. You can get the location of the current virtualenv with the following command: ``` pipenv --venv ``` ``` $ pipenv --venv C:\Users�cer\.virtualenvs\pipenv-blog-gHY6vF9t ``` For Windows, it is in the Admin user(in my case it is named `acer`) followed by the hidden folder `virtualenvs`, this folder will contain all the virtualenvs for different projects using `pipenv`. If you wished to change this location and keep the virtual environment folder in the same directory as your project, you can set up the environment variable for it as follows: For Linux/macOS: Add the following to your `~/.bashrc` or other equivalent shell configuration file. ``` export PIPENV_VENV_IN_PROJECT="enabled" ``` For Windows: Add it to the PATH Environment variable. ``` set PIPENV_VENV_IN_PROJECT="enabled" ``` This will make sure the virtualenvs for the project in `pipenv` are created inside the current folder itself and not in a single `~\.virtualenvs\` folder. ## Creating the requirements.txt file Let's say you also want to create a requirements.txt file for distributing your project to someone else, as not everyone will use Pipenv to manage their dependencies. It is really straightforward and quick to create a requirements.txt file from the Pipenv environment. The below command will make the `requirements.txt` file from the existing Pipenv project. ``` pipenv lock -r >requirements.txt ```  This will create the requirements.txt file, based on the Pipfile. Pipenv is smart again to provide all of the required dependencies to the requirements.txt in order that pip will be able to install all the required dependencies. ## Using requirements.txt in Pipenv We can install all the dependencies from the requirements.txt file while we are migrating from bare-bones virtualenv and pip to Pipenv. Pipenv will install all the mentioned dependencies and it will also add its checks for the appropriate checks for dependencies. ``` pipenv install -r requirements.txt ``` This will install the dependencies mentioned in the requirements.txt file into the Pipenv virtual environment. ## Managing Dev Packages Let's say we want to install a package but we are not sure to use it in production or the actual script, just a test for how it will work. Thus we have dev packages to install optional or testing packages. To install a dev-dependency or package, you can install via the following command: ``` pipenv install <package-name> --dev ```  If we see the Pipfile, the `django` package that we installed tagged with `--dev` will be in the `dev-packages` ``` $ cat Pipfile [[source]] url = "https://pypi.org/simple" verify_ssl = true name = "pypi" [packages] flask = "*" [dev-packages] django = "*" [requires] python_version = "3.8" ``` If we wanted to uninstall all the dev-packages, we can simply enter the command: ``` pipenv uninstall --all-dev ```  This will simply uninstall all the dev dependencies/packages from the pipenv environment and also remove them from the packages list in Pipfile. If you wished to uninstall a specific package in pipenv, you can uninstall it by the simple command: ``` pipenv uninstall <package-name> ```` ## Installing/Uninstalling all packages To install only the default packages and not dev-packages. ``` pipenv install ``` To install or configure a project, if you want to test the project with all the dev dependencies, you can install them with: ``` pipenv install --dev ``` This will install all the packages both `packages` and `dev-packages`. If you want to uninstall all the packages in pipenv, you can use the command : ``` pipenv uninstall --all ``` This will uninstall all the default and dev packages from pipenv. This is like starting a fresh virtual environment. ### References: - [Pipenv documentation](https://pipenv.pypa.io/en/latest/) - [Real Python - Pipenv Guide](https://realpython.com/pipenv-guide/) ## Conclusion So, this was a brief introduction to `pipenv` which is Python's recommended package management tool. We saw that we have control over the dependencies and packages in a virtual environment which is taken care of automatically for us. We don't have to activate the environment to install or manage any package for a project. https://www.meetgor.com/posts/podevcast-project Introduction Hello Developers! Want to listen to programming podcasts from a single place? Podevcast is the place you should be searching for. I am Meet Gor and Fri, 18 Feb 2022 00:00:00 UTC ## Introduction Hello Developers! Want to listen to programming podcasts from a single place? Podevcast is the place you should be searching for. I am Meet Gor and I present this project as a submission to the Netlify x Hashnode Hackathon. Podevcast is a webpage(static) for listening to podcasts centered around developers and programming. Just pick your favorite one and start listening straight away. Let's dive into the making of Podevcast. Head on to https://podevcast.netlify.app/ to check out the live app. ## What is Podevcast Podevcast is a web application or a static site that renders the top programming/development podcasts. You can listen to the top podcasts around the developer community from a single source. > Listen to your favorite developer podcasts with Podevcast Podevcast is a static site generated using a script. There is a static site generator that is heavily done in Python and deployed to Netlify. You can simply listen to the podcasts on the web page or go to the canonical page of the podcast episode. From the canonical page, you can choose to hop to your chosen music player, but the default music player should be fine for casual listening. The core idea is to keep things in a single place for developer podcasts. [Podevcast](https://podevcast.netlify.app/) [Source Code](https://github.com/Mr-Destructive/podevcast) ## Preview Podevcast has popular developer podcasts like `Command Line Heroes`, `The Python Podcast`, `The freeCodeCamp Podcast`, and many others to choose from. You can go into categories for looking at a specific podcast. ### Application Demonstration Here's a small demonstration of the Podevcast application.  Podevcast has multiple pages like: 1. [Home page](https://podevcast.netlify.app/) 2. [Podcast page](https://podevcast.netlify.app/list) 3. [Episode page](https://podevcast.netlify.app/the_real_python_podcast/ep/1/) 4. [Podcast List page](https://podevcast.netlify.app/command_line_heroes/) 5. [Categories page](https://podevcast.netlify.app/category/) The Home page has the latest episode of all the podcasts. It also has an audio player to play on the go.  The Podcast List page has the list of all the Podcasts available in the project. It has the name of the podcast with the link to the podcast page that has the list of all the episodes of that podcast.  The categories page has a list of categories of the podcasts like Web-development, backend, frontend, data science, DevOps, and so on. More categories will be added soon.  The Episode page has the audio player, the summary of the episode, canonical episode, and podcast page.  ## Why Podevcast? Listening to music is one thing and listening to podcasts is different. I wanted a place from where developers can listen to developer-specific podcasts from a single source not just give out the article **"Top 10 podcast you should be listening to as a developer"**. Having played around with python and some libraries like feedparser and jinga previously I saw this Hackathon as an opportunity to convert the idea into a project. It fits the JAMStack area well from the Hackathon and project perspective. ## Tech Stack - Python - [feedparser](https://pypi.org/project/feedparser/) - [jinga2](https://pypi.org/project/Jinja2/) - GitHub Actions - HTML / CSS The data is extracted from various RSS Feeds using the feedparser library in Python. Using GitHub Actions, the feed is refreshed every 24 hours to fetch the latest episodes from the respective podcast feeds. Basically, the GitHub action triggers a Netlify deployment that in turn generates the static site by running the script. The command for running the script on Netlify and generating the `Podevcast` webpage is : ``` pip install -r rquirements.txt && python script.py ``` And the directory for deployed web pages (published directory) is `site` which contains all the `HTML` files that can be rendered as the website itself. ### Source Code The project is available on [GitHub](https://github.com/Mr-Destructive/podevcast). Feel free to open a PR to add a Podcast or a Category. The project only has a few python files, the main script is `script.py` which actually creates the home and the podcast list pages along with the episode pages. The `src` folder contains some extra bits of scripts like creating the categories and category podcast list pages. Also, it has certain config files like `runtime.txt`, `requirements.txt`, and so on. Finally, there is the `podlist.json` for the list of podcasts and `categorylist.json` for the categories of podcasts. ### Core Script Snippet The python script looks a lot bigger than the below snippet but it is doing the same process multiple times for different pages. There is also some type checking and tiny details that are added as per the requirement of the templates. ```python import feedparser from jinja2 import Environment, FileSystemLoader from pathlib import Path template_env = Environment(loader=FileSystemLoader(searchpath='./layouts/')) index_template = template_env.get_template('index.html') episode_template = template_env.get_template('episode.html') feed = feedparser.parse("https://freecodecamp.libsyn.com/rss") pod_name = feed['feed']['title'] for i in range(0, len(feed['entries']): ep_title = feed['entries'][i]['title'] audio = feed['entries'][i]['links'][1]['href'] cover_image = feed['entries'][i]['image']['href'] og_link = feed['entries'][i]['links'][0]['href'] episode_obj = {} episode_obj['title'] = ep_title episode_obj['audiolink'] = audio episode_obj['cover'] = cover_image episode_obj['link'] = og_link with open(os.path.join(f"site/{pod_name}/ep/{i}/index.html"), 'w', encoding='utf-8') as ep_file: ep_file.write( episode_template.render( episode = episode_obj ) ) ``` Above is a simple snippet of the core functionality of the script. It basically takes the RSS Feed `URL` of the podcast and using `feedparser` the data is retrieved in the form of a dictionary in Python. - Iterate over the `feed['entries']` which is a list of lengths same as the number of episodes of that podcast, we then assign a set of values like `episode title`, `audio link`, `cover image`, `canonical link for the episode`, `date` and so on. - Create a dictionary and store the key value as the mentioned data to access from the template. - Open a file in the structured file format and then parse the `episode_obj` which is a dictionary to the episode template. - Access the dictionary using jinga2 templating tags. ```html <html> <head> <title>Podevcast</title> </head> <body> <h3 class="ep-title">{{ episode.title }}</h3> <img src="#"> <a class="ep-link" href="#">Episode </a> <audio controls="enabled" preload="none"> <source src="#" type="audio/mpeg"> </audio> </body> </html> ``` We can use `{{ }}` to access any value parsed to the template via the script. Also, we can make use of `{% %}` to run loops, conditionals, blocks, and other tags in the template.  So, we can see the feed is basically a dictionary that has a key-value pair and further, it can be a nested dictionary or a list as a value of a key. As in the case of `feed['entries']` is a list with the length of the number of episodes of a podcast. And in the script, we use various keys to access various components, obviously, this requires a bit of exploration of the dictionary initially but it becomes easy thereafter to automate using Python. ### Episode List Currently, the episodes are added using the JSON file. It is not that user-friendly but still not a big task to simply add a link in a file. This is a #TODO that will require some external tooling to integrate into the webpage to ask for a form to submit a new Podcast. ```json { "Command Line Heroes": "https://feeds.pacific-content.com/commandlineheroes", "Python Podcast__init__": "https://www.pythonpodcast.com/feed/mp3/", "Real Python Podcast": "https://realpython.com/podcasts/rpp/feed", "The freeCodeCamp Podcast": "https://freecodecamp.libsyn.com/rss", "CodeNewbie": "http://feeds.codenewbie.org/cnpodcast.xml", "Linux For Everyone": "https://feeds.fireside.fm/linuxforeveryone/rss", "JavaScript Jabber" : "https://feeds.fireside.fm/javascriptjabber/rss" } ``` The process requires a manual test to validate a given RSS Feed as not all feeds are generated the same way and thus there are a few exceptions that need to be sorted out manually. For example, the Python Podcast doesn't have a cover image parsed into the RSS Feed, so there needs to be a check for it in the script and also in the template to restrict parsing and displaying the cover image link. ### Episode Categories This is also a JSON file that holds the keys as the category and the value as a list of episode names (strictly the name from `feed['feed']['title']`). There needs to be a human decision to be taken to add the podcast into a specific category. ```json { "Python":[ "Talk Python To Me", "The Python Podcast.__init__", "The Real Python Podcast", "Python Bytes" ], "Javascript":[ "Full Stack Radio", "JavaScript Jabber" ], "Linux":[ "Command Line Heroes", "LINUX Unplugged", "The Linux Cast", "Linux For Everyone" ], "Data Science":[ "DataFramed", "Data Skeptic", "The Banana Data Podcast" ], "Dev Ops":[ "DevOps Cafe Podcast", "Arrested DevOps", "Pulling the Strings", "Azure DevOps Podcast", "DevOps and Docker Talk" ] } ``` Though the JSON file is managed manually the generation of the categories is automated. Please feel to add other categories of your choice. ## What's Coming? Certain features like adding podcast using a form, adding more podcasts, and categories for sure. Though what looks a bit cloudy in my opinion is adding accessibility links to music players because the RSS feed doesn't contain direct links to them. Though I still to explore and find out if it can be obtained from the feed itself. - Search box for searching podcasts - Accessible Links to other platforms (Spotify, Itunes, etc) - More depth in categories (Languages/Frameworks/Niche-specific podcasts) I'll add these features after checking the feasibility of the ideas and the response from the community after releasing them. ## Final Words This project wouldn't have existed without this Hackathon as it gives a deadline to finish and hope to win something. Specially thanks to Hashnode and Netlify for organizing such a great opportunity in the form of a hackathon. Also, the maintainers of Python libraries like feedparser and jinja. The project would have been impossible without them. If you like the project please give it a star on [GitHub](https://github.com/Mr-Destructive/podevcast). Have any feedback? Please let me know in the comments or on [Twitter](https://twitter.com/MeetGor21). Thank you for reading, Hope you have a good time using Podevcast. Happy Coding :) https://www.meetgor.com/posts/django-basics-crud-api Introduction After understanding the admin section and setting up a database, we can finally move on to the CRUD API in Django. We will create a API in pure Dja Thu, 03 Feb 2022 00:00:00 UTC ## Introduction After understanding the admin section and setting up a database, we can finally move on to the CRUD API in Django. We will create a API in pure Django with Model Form, Class Based Views and templates. We will go in two parts in this section, first revising the workflow of the app creation and setup, the next part all about CRUD API. We'll dive into a lot of familiar topics and concepts from previous parts but there are is still a lot to learn beyond the basics. The following is the demonstration of the CRUD API we will be making in this section:  ## Quickly Setting up an app We have seen how to create an app in the [folder structure part](https://mr-destructive.github.io/techstructive-blog/django/python/web-development/2021/11/24/Django-Basics-P3.html) of the series, after that we have created all the components namely the views, urls, templates, etc in distinct parts. In this article, we will create an app together in a single part. We'll discuss all the process in short because its important to get the workflow of Django to ease the actual coding process. Let's dive in the part one of this section of creating a CRUD API i.e. to set up an app. ### Creating App To quickly set up an application, we need to execute an command so that python under the hood creates a folder for an app in the current Django project. ``` python manage.py startapp api ``` This will set up an folder `api` in the base directory of current django project. Now, we'll need to configure the Django settings for the project to pick up this app while running the server, making any migrations or any other project level process. ### Registering the App-name in settings Simply append the name of the app in a string in the `INSTALLED_APPS` list inside the `settings.py` file. ```python # project_name / settings.py # Application definition INSTALLED_APPS = [ 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'api', ] ``` ### Setting up the URLS We need to create a `urls.py` file for the `api` app and link it in the main project's URLs. ```python # app-name / urls.py from django.urls import path urlpatterns = [ ] ``` We'll keep the url patterns empty but after setting and configuring the project and the application, we'll map the routes to the views in this app. Next, we need to update the `urls.py` file in the project folder to include the `api` routes/urls. I have kept it '' or base route, it could be anything as per your application design. ```python # project_name / urls.py from django.contrib import admin from django.urls import path, include urlpatterns = [ path('admin/', admin.site.urls), path('api/', include('app_name.urls')), ] ``` After configuring the URLs we need to set up the templates and static files. ### Setting up Templates and Static files To set up the templates, we need to configure the `settings.py` file to look for the templates in a specific directory. Below is the snippet to change the default configuration. ```python # project_name / settings.py import os TEMPLATES = [ { 'BACKEND': 'django.template.backends.django.DjangoTemplates', 'DIRS': [os.path.join(BASE_DIR, 'templates'),], 'APP_DIRS': True, 'OPTIONS': { 'context_processors': [ 'django.template.context_processors.debug', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.contrib.messages.context_processors.messages', ], }, }, ] ``` So, you can now create your templates (HTML documents) inside the `templates` folder after creating the folder in the base project directory. TO configure static files, we need to also make modifications to the `settings.py` file as follows: ```python # project_name / settings.py # import os STATIC_URL = '/static/' STATICFILES_DIRS = [os.path.join(BASE_DIR, "static")] STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles') ``` Here, we are configuring the static files ( CSS, Js, Assets) in the `static` folder. This can be anything you like but the folder name should be then changed accordingly. Similar to the Templates folder, the static folder is also located in the root directory of the project. This is all the necessary configuration for simple full stack application you can make, still we have to configure the backend and write the actual logic for the application. We'll move on to the second part of this section i.e. creating the CRUD API. ## Making an CRUD API There are certain things to be planned before we move ahead like the selection of database, the schema of the database and basically the structure of the backend. For now, we'll go with SQL database, a REST API with PostgreSQL hosted locally on the system. ### Configure the database To configure the database, we simply need to first create the database. This can be done by using a CLI or the Admin interface of the particular database we are working with in our case it's PostgreSQL. Postgres comes with pgAdmin to create and manage the databases and the server locally. The detailed explanation of the [database creation](https://mr-destructive.github.io/techstructive-blog/django/python/web-development/2022/01/16/Django-Basics-P8.html) is explained in the previous parts of the series. After creating the database locally, you need to tweak the `settings.py` file database configuration object as per your credentials of the database instance. ```python # project_name / settings.py DATABASES = { 'default': { 'ENGINE': 'django.db.backends.postgresql', 'NAME': 'blogapp', 'USER': 'postgres', 'PASSWORD': '@1234567', 'HOST': 'localhost', 'PORT': '5432', } } ``` If you have already created the databse, there is no problem in reusing the same one but for production level applications make sure to keep things separate. Also the model name has to be distinct in each app to create tables in the database. ### Creating Model We can now finally move on to the logic of the application, this is the part which acts as the backbone of backend development in django. You need to design the model carefully and programmatically. Make sure to include most of the logic as you can inside the models as it is a standard and a good practise to develop in professional projects. For this app, we simply are going to create a blog post model. We are going to have some simple attributes and structure. ```python # app_name / models.py from django.db import models from django.contrib.auth.models import User class Article(models.Model): title = models.CharField(max_length=127, verbose_name="headling") post = models.TextField(verbose_name='content') author = models.ForeignKey(User, on_delete=models.CASCADE) created = models.DateTimeField(auto_now_add=True) updated = models.DateTimeField(auto_now=True) def __str__(self): return self.title class Meta: verbose_name_plural = 'Articles' ``` This is again a simple model for demonstration of a lot of things like the `str` function, `Meta` class, `verbose_names` and so on. You can design models as per your requirements and proper planning. It really determines the structure of an application. Now, after we have created the models, we need to migrate the schema into the database. Remember it acts as a checkpoint in the history of changes to the database. ``` python manage.py makemigrations ``` This will create a checkpoint in the `migrations` folder in the app. To make changes to the database, we use the migrate command. ``` python manage.py migrate ``` Here's a quick demonstration of everything covered so far.  ### Creating Form Django has a functionality to convert the fields in a model to a form which can be used for input in the frontend side. This allows us to just configure the Form and map the form to a view for the specific operation and simply add a tiny snippet of HTML to make a basic API and thus handle the entire heavy lifting. [Model Forms](https://docs.djangoproject.com/en/4.0/topics/forms/modelforms/#modelform) are the type of forms in django that does the above mentioned functionality. We need to create a form class in a new file inside the app called `forms.py` and add the following content: ```python # app_name / forms.py from django import forms from .models import Article class ArticleForm(forms.ModelForm): class Meta: model = Article fields = [ 'title', 'post', ] ``` In this snippet, we are creating a Form called `ArticleForm` it can be anything but (Model_NameForm) is a standard way to name a Model Form. It is a child class of the ModelForm, i.e. it inherits the parent's attributes/properties and methods. So we have the access to it's Meta class which defines some of the attributes like model, fields, etc. Here, we are only adding `title` and `post`, because `created` and `updated` are automatically added. So what about `author`? Well, we can add it automatically by processing the request that will be sent when the form is submitted (we'll discus while creating views). In a model form, we also have other attributes like `exclude` which is the opposite of `fields`, say you have a lot of attributes in a model and you want to skip certain fields then you don't use fields and use exclude to specify only the attributes to be excluded. Also `widgets` which can be used to style and properly process the fields in the form to have more control on how the form should be validated and presented. ### Updating Models Now, when we have chosen to use Model Form, we need to update the model logic (not the structure so no migrations). We'll have to add some methods and redefine certain default parameters in able to use the Class based views and Model forms to the fullest strength. ```python # app_name / models.py from django.db import models from django.contrib.auth.models import User from django.core.serializers import serialize import json class ArticleQuerySet(models.QuerySet): def serialize(self): list_value = list(self.values("id","author","title","post")) return json.dumps(list_value) class ArticleManager(models.Manager): def get_queryset(self): return ArticleQuerySet(self.model,using=self._db) class Article(models.Model): title = models.CharField(max_length=127, verbose_name="heading") post = models.TextField(verbose_name='content') author = models.ForeignKey(User, on_delete=models.CASCADE) created = models.DateTimeField(auto_now_add=True) updated = models.DateTimeField(auto_now=True) objects = ArticleManager() def __str__(self): return self.title def serialize(self): data = { "id": self.id, "title": self.title, "author": self.author.id, "post": self.post, } data = json.dumps(data) return data class Meta: verbose_name_plural = 'Articles' ``` Let's break down what was added: - Created two new classes (`ArticleManager` and `ArticleQuerySet`) This was done to manage and serialize the model attributes. We need a standard like JSON to parse and return the data from the client to the server and vice versa. The `ArticleQuerySet` actually converts the `QuerySet` into list and dumps into a json object. **A QuerySet in Django is collection of SQL queries.** - Function serialize The serialize function actually converts the `QuerySet` into a JSON object which in turn is used as a utility function for the model. **It returns the JSON object hence a serialized object from a queryset.** - Editing the object for the model The object for the model is re initialized as a object of the `ArticleManager` class which in turn manages the and process the query set and returns the serialized object. That's all done from the models for the app and now we finally move to the views. ### Creating Views Now, we can start writing the views to actually add the CRUD functionality ourselves and slowly develop the frontend as well. To start creating views, we need to import some built-in classes used for making APIs like the Django generic views. Some of the core views(classes based) for making a CRUD API include: 1. [CreateView](https://docs.djangoproject.com/en/4.0/ref/class-based-views/generic-editing/#createview) 2. [UpdateView](https://docs.djangoproject.com/en/4.0/ref/class-based-views/generic-editing/#updateview) 3. [DeleteView](https://docs.djangoproject.com/en/4.0/ref/class-based-views/generic-editing/#deleteview) 4. [DetailView](https://docs.djangoproject.com/en/4.0/ref/class-based-views/generic-display/#detailview) 5. [ListView](https://docs.djangoproject.com/en/4.0/ref/class-based-views/generic-display/#listview) These views help in making the API in Django easily. We simply need to add the template name to add a customize the layout, make forms, define the sets of fields to input from the client and which fields to process by the server side. Let's create them one by one and understand the structure of class based views. #### Create View This view as the name suggests is used for creating a entry in a model(database) or we can also say that it will act as the `Create`(POST request) in the CRUD API. ```python # app_name / views.py from django.views.generic.edit import ( CreateView, UpdateView, DeleteView ) from .models import Article from .forms import ArticleForm class ArticleCreateView(CreateView): model = Article form_class = ArticleForm success_url = '/api/' def form_valid(self, form): form.instance.author = self.request.user return super(ArticleCreateView, self).form_valid(form) ``` Here, we have created a Class based view called `ArticleCreateView` which inherits the `CreateView` class from the `django.views.generic.edit` module. Here as similar to the `ArticleForm` class, we have certain attributes to pass like the model, form class and the success url. - The `model` simply indicates to which table or model we are creating the view for. - The `form_class` denotes the ModelForm class we are using for the Create view. - The `success_url` is the url route to which to redirect after a successful POST request/form submission. Remember we discussed about adding `author` field automatically from the request from the form. This process is carried out in the `form_valid` function. [Form_valid](https://docs.djangoproject.com/en/4.0/ref/class-based-views/mixins-editing/#django.views.generic.edit.FormMixin.form_valid) is a helper built-in function to redirect to the success_url when the form data is being posted. Before actually doing that, we add the author field by setting it as the User by accessing the `self.request` object. The `self.request` object actually holds the meta-data about the request that is being sent to the API so we can access the User who is currently logged in. Though we don't have a login system, we can assume the user is logged in with some user account. Now to handle certain exceptions we can add [Mixins](https://docs.djangoproject.com/en/4.0/topics/auth/default/#django.contrib.auth.mixins.AccessMixin). We can use [LoginRequiredMixin](https://docs.djangoproject.com/en/4.0/topics/auth/default/#django.contrib.auth.mixins.LoginRequiredMixin) to only allow the form submission for logged in users and so on. #### Update View ```python # app_name / views.py class ArticleUpdateView(UpdateView): model = Article form_class = ArticleForm success_url = '/api/' def form_valid(self, form): form.instance.author = self.request.user return super(ArticleUpdateView, self).form_valid(form) ``` This is quite similar to the `CreateView` except we have to use `UpdateView` as the base model of the `ArticleUpdateView` the rest of the attributes remain the same and functioning of the form like pre-rendering the fields to edit are all managed by the `UpdateView` out of the box. #### Delete View ```python # app_name / views.py class ArticleDeleteView(DeleteView): model = Article form_class = ArticleForm success_url = '/api/' def form_valid(self, form): form.instance.author = self.request.user return super(ArticleDeleteView, self).form_valid(form) ``` Again, here we do not have to change anything as for attributes and the `valid_form` function. We'll see the details when we look at the templates. #### List View ```python #app_name / views.py class ArticleView(ListView): model = Article template_name = 'api/list.html' def get_context_data(self, **kwargs): context = super().get_context_data(**kwargs) context['articles'] = Article.objects.filter(author=self.request.user) return context ``` This view is for listing all the objects/articles for the current logged in author. If we want something like a homepage for all the articles, the thing is as simple as replacing ```python context['articles'] = Article.objects.filter(author=self.request.user) ``` by ```python context['articles'] = Article.objects.all() ``` You can even apply filters for fetching the latest post by `Article.objects.filter().order_by("-created")`. You get the idea. It boils down to simple python and library function. Also, we have a new attribute `template_name` which allows us to use the data in our custom template. In this case we will create the template in the `templtes/api/` folder called `list.html`. The function `get_context_data` is used to fetch the objects from the database and return it as a special dictionary(JSON object) called `context` which can be rendered in the templates by the key in this case, the `articles` key will give us all the articles by the current user. #### Detail View ```python # app_name / views.py class ArticleDetailView(DetailView): model = Article template_name = 'api/post.html' ``` As opposite to the ListView, we have the DetailView that renders the details of the particular article. Here we don't have to write the `get_context_data` function as the default object for accessing the model data is `object`. So, we simply need to specify the `template_name` and the `model`. Here, the template is `api/post.html` in the templates folder. ### Mapping the URLS After completing the views, we can map those into a urls to access them as endpoints in the api app. Let's build upon the `urls.py` file that we created with no url paths. ```python # app_name / urls.py from django.urls import path from .views import ( ArticleView, ArticleDetailView, ArticleCreateView, ArticleUpdateView, ArticleDeleteView ) urlpatterns = [ path('', ArticleView.as_view(), name="listpost"), path('article/<pk>/', ArticleDetailView.as_view(), name="detailpost"), path('create/', ArticleCreateView.as_view(), name="createpost"), path('update/<pk>/', ArticleUpdateView.as_view(), name="updatepost"), path('delete/<pk>/', ArticleDeleteView.as_view(), name="deletepost"), ] ``` So, we can see the views are imported from the .views and are mapped to a particular route. We use `.as_view` function to take a request from a callable class based view and returns the processed response. The `<pk>` is the id for accessing a particular object. We use the id for detail view, update and delete views. ### Creating Templates and Static files We now, have to join the final piece of the puzzle i.e. to create templates. It's quite simple but requires a bit of logic to connect things together. The backend is handled flawlessly by Model Form and Generic views, we simply need to put simple HTML and Django templating language to its correct use. ```html # templates / index.html <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>Django Blog</title> {% load static %} <link rel="stylesheet" href="#"css/style.css' %}"> </head> <body> <h1>Articles</h1> {% block body %} {% endblock %} </body> </html> ``` A simple HTML doc with link to a stylesheet. We also have a block to further inherit this as the base template. ```css # static / css / style.css body { background-color:#1d1dff; color:white; } h1 { text-align:center font-family: monospace; } a{ text-decoration-color: #00ffff; color: #ff6600; } p { color:#ff6600; font-weight:500; } ul { list-style-type:square; } ``` The below template is a detail view page, that renders the details of the article. We are using the `object` key from the `context` dictionary provided by default as mentioned [here](). We are also embedding the `Update` and `Delete` buttons to manage the Article. This is the time to explain about the dynamic urls in Django. We have used the `{% url 'updatepost' object.id %}` to create a dynamic url that will redirect to the `updatepost` url and parse with it the id of the object. This is important as the url itself takes the `<pk>` value to update the post. So we just pass the id like a parameter to the URL. This is the way we create dynamic urls in Django templates which is quite similar to static file urls. ```html # templates / app_name / post.html {% extends 'index.html' %} {% block body %} <h2>{{ object.title" }}</h2> <p>{{ object.post" </p> <button type="submit" onclick="window.location.href="#"updatepost' object.id %}'"> Update </button> <button type="submit" onclick="window.location.href="#"deletepost' object.id %}'"> Delete </button> {% endblock %} ```  We have used the dynamic URLs in Update and Delete View buttons. The below template is for creating a home page like structure but for the current logged in user. We are displaying all the articles of the logged in the form of a list with a for loop as discussed in the templating tutorial part. We have access to the key `articles` from the dictionary `context` and hence we iterate over the loop from that object and display the title with the url for the post detail view. Here as well we are using the dynamic url by parsing the `article.id` as the `<pk>` primary key to the url. ```html # templates / app_name / list.html {% extends 'index.html' %} {% block body %} <ul> {% for article in articles %} <li><a href="#"detailpost' article.id %}">{{ article.title }}</a></li> {% endfor %} </ul> {% endblock %} ```  ```html # templates / app_name / article_confirm_delete.html {% extends 'index.html' %} {% block body %} <form method = "post"> {% csrf_token %} <p> Are you sure to delete Post:<b> {{ object.title }}</b> ? </p> <input type="submit" value="Delete"/> {% endblock %} ```  ```html # templates/ app_name / article_form.html <form method="POST" enctype="multipart/form-data"> {% csrf_token %} {{ form.as_p" }} <input type="submit" /> </form> ``` This is the template that forms the basis for all the forms like Create and Update. We use the template variable `{{ form }}` to render the form in the template. Additionally we use `.as_p` to make the form fields as a paragraph tag in HTML. The `{% csrf_token %}` is the Cross site forgery token allowing secure posting of a form. Don't bother about it much it is important for identifying the user posting the data from the site. The final piece of element is the submit button used to submit the form.   For any references, you can take a look at the [GitHub](https://github.com/Mr-Destructive/django-blog) repository for the CRUD API in pure Django. ## Conclusion So, from this section, we were able to create a CRUD API in pure Django. This should be the end of the core series of Django Basics, but there are more parts coming up still which are not basics but still worth learning as a beginner. We'll further explore DRF, Forms, CORS, etc. We'll see some other libraries specific for Django. Hopefully from this tutorial series, you were able to learn something and if you have any queries or issues please let me know. Thank you for reading, please provide feedback on how you felt about the series. Though the series is not over, it's sufficient for a beginner to get pace in Django. Hope you had a great time. Happy Coding :) https://www.meetgor.com/posts/django-basics-admin Introduction In the previous section, we configured the database in our Django project. We will move ahead and interact with the Admin Section in Django. Django Wed, 26 Jan 2022 00:00:00 UTC ## Introduction In the previous section, we configured the database in our Django project. We will move ahead and interact with the Admin Section in Django. Django has a built-in Admin Section which we can use to manage our apps and models. We will create a admin account(superuser) and try to render our custom Model in the Admin Interface. We shall further discuss the customization in the Admin section as per our requirements and necessity. ## What is the Admin Section? Admin Section is a powerful built-in utility provided by Django. It gives the administrative rights over the web project, the interface is neat and provides out of the box functionality to interact with the models in our project without us manually creating any mapping the views and urls. It is restricted to only superusers or trusted users to use as it is for administrative purpose. The Admin section is present by default for any django application. The interface provides the Django User and Group Model by default. Additionally we can have our own custom models to interact with. For every registered model you have the CRUD (create / read / update / delete ) functionality which makes it very easy and convenient to test the working of model before working around with APIs or moving ahead in the project. ## Setting up an admin account (superuser) In order to access the Admin section, we need to create a superuser. A superuser as the name suggests is a user who has the supreme authority for performing operations in the project in this case a web application. To create a super user we need to run a command from the command line that takes our name, email and password as input to create the super user. ```bash python manage.py createsuperuser ``` This will prompt you for a couple of things like : - `username` the default is `admin`. - `email` it's not necessary to put one. - `password` should be at least eight characters long The password input will be silent which means you cannot see what you type for security reasons, and the password field will be confirmed once, so you'll have to enter the password once more. But that's all you have to do to create a super user in Django for your web project.  ## Navigating the admin section After creating a superuser, we can now navigate the admin section from the browser. The admin section is by default located in the `/admin` url-route i.e. you need to navigate to `http://127.0.0.1:8000/admin` here `8000` can be any port as your preferred port number for the django application. After vising the Admin route, you will be prompted to a Login Screen. You simply need to add in the username and password which you entered while creating the superuser a while ago and you should be in the Admin Section. The default admin section as of Django `3.2.9` looks like following:  ## Components of the Admin Section The Admin interface looks quite simple but is quite powerful and customizable. We have the Default Models in Django which are labelled in the `Authentication and Authorization` section namely the `Users` and `Groups`. You can see we have `+ Add` and the `Change` links to actually create the data associated with those Models. In the Admin Section you can basically play around with your models, it was not designed to act as the frontend for your application that's what the Django [documentation](https://docs.djangoproject.com/en/4.0/ref/contrib/admin/#module-django.contrib.admin) says and is absolutely correct. So, we don't have much things to explore when the UI is concerned as it is simple and straight forward to understand. We'll dive into how to register our models into the Admin section and from there on we can explore the UI to perform CRUD operations. ### Built-in Models Django has two built-in and registered models in the Admin Section as said earlier. 1. Users 2. Groups Users is basically the Django's User Model which provides the basic Authorization functionalities which further can be added to the we application. The Super User that was created from the `createsuperuser` command was associated with the Django User model. We have basic fields in the User model like: 1. Username 2. Email-ID 3. Password If we go to the route `http://127.0.0.1:8000/admin/auth/user/add/`, we can see a form like UI that allows us to add a User.  But there is no option for the `Email-ID` that's because for the User model has been modifies from the base [User Model](https://docs.djangoproject.com/en/4.0/ref/contrib/auth/#user-model) and we can see the `superuser` has all the attributes the `User` class has like the email, first name, last name and so on.  We can even extend this functionality or modify the existing attributes of the User Model in our custom Model. For example, we can add Age, Phone number, etc in to our Custom User Model. How? We'll look into that later but that is to give an idea about the User Model. There is a lot more than just the details like Username, email and password. We need a way to manage which user is allowed to access what components. So, this is termed as `Permissions` in the User model, for the super user we might have the access to the admin page but a regular User might not. This is a permission which is by default implemented by Django in the User model. Like wise we can extend this functionality to add more permissions depending on the Model we are working with. ### Groups This is the model which can hold a group of certain Model. The typical example hers is a group of User and its permissions. We can have a group for the developers of a project(this project) and a separate Group for rest of the Users. This creates a well-defined boundary for different types of User in a larger application. Currently, we don't have any groups created by default. Since it is a concept to be learnt for a quite large project with thousands of Users. ## Registering Models in Admin Section Now, what are the default Admin section looks like, we can move on to register our own models in the Admin section. To do that, inside the app folder(a django app) you will see a `admin.py` file. Make sure to be in the app in which you have created a model. We need to register a Model to the admin section. ```python # app_name/admin.py from django.contrib import admin from .models import Article admin.site.register(Article) ``` The `admin.site.register` basically adds a Model to the Admin Interface. The article Model is defined as follows: ```python # app_name/models.py from django.db import models class Article(models.Model): title = models.CharField(max_length=127) post = models.TextField() created = models.DateTimeField(auto_now_add=True) updated = models.DateTimeField(auto_now=True) ``` This will add the Model `Article` in the Admin Section. We can now perform CRUD operations in the Admin section. ## Performing actions with Admin Section So, perform CRUD operations, we can navigate to the Model Article and simply click the `Add Article` Button to add a object of the model Article. We will be presented a form to fill. Now here we, can see the fields which are actually to be inputted by the user. We don't see the fields like `created` and `updated` as they are automatically set as per the current time.  After filling this form you will see `Article object (1)` which looks like a non-sense thing to look at. This is where the `__str__` [function](https://docs.djangoproject.com/en/4.0/ref/models/instances/) comes handy and saves the day.  Now, we can see we also have the ability to Delete and Update the Post. This is the best thing about the Django admin interface. It's quite intuitive for complex model to be tested before moving ahead and making necessary corrections if required. ### Modifying the Model (without migrations) Now, at this stage if we forgot to add the `__str__` function, we need to add it into our models. But what about the migrations? We do not need to migrate this changes as there is no change in how to model is structures. We are changing how to Admin Interface should present our model. So, we'll make changes to our model but only aesthetically. ```python #app_name/models.py from django.db import models class Article(models.Model): title = models.CharField(max_length=127) post = models.TextField() created = models.DateTimeField(auto_now_add=True) updated = models.DateTimeField(auto_now=True) def __str__(self): return self.title ``` And now if we refresh the Admin page, we can see the title is the object name :)  This looks a trivial change but makes a big difference for large applications and dataset. ### Verbose name We can add a verbose name for an attribute inside an Model. A verbose name is a human readable name for a field. So, let's say we have a attribute/field called `fname`, the person who might have created the model might know it stands for `first name` but someone else might not. So in this case, we can add the `verbose_name` to be used as the name in the Admin Section. We also have `verbose_name_plural` which will be handy for a model name. We do not wnat Django to just add `s` before any Model name, it might look good for `Articles`, `Questions`, `Posts` but for `Quizs`, `Categorys`, `Heros` look too funny than `Quizzes`, `Categories` and `Heroes` respectively. ```python # app_name/models.py from django.db import models class Article(models.Model): title = models.CharField(max_length=127, verbose_name="headline") post = models.TextField(verbose_name='content') created = models.DateTimeField(auto_now_add=True) updated = models.DateTimeField(auto_now=True) def __str__(self): return self.title ```  Again, we do not need to migrate any changes to the database as it is not a logical change in the schema of the database. ```python # app_name/models.py from django.db import models class Article(models.Model): title = models.CharField(max_length=127, verbose_name="title") post = models.TextField(verbose_name='content') created = models.DateTimeField(auto_now_add=True) updated = models.DateTimeField(auto_now=True) def __str__(self): return self.title class Meta: verbose_name_plural = 'Articless' ```  Though `Articless` sounds weird, it is just made for realizing that Django by default adds `s` to the Model name for representing as a Class in the Admin. ### Admin Register Class We can even list not only the title but many things in a particular format, we need to define a class which will be derived by the `admin.ModelAdmin` class. ```python # app_name/admin.py from django.contrib import admin from .models import Article @admin.register(Article) class Article(admin.ModelAdmin): list_display = [ 'title', 'created', 'updated', ] ```  We are using the `admin.ModelAdmin` class as the base class to overwrite the `list_display` list which will display the mentioned items in the Admin Section as a list. We are also using the `admin.register` as the class Decorator to actually register the model in the Admin section. Now, we can see a lot of customization in the Admin section. This is just a glimpse of what customization is. This can be tailored as per your model and needs. ## Conclusion So, in this section we were able to interact with the Admin section in Django. We were able to register our custom Model in the Admin interface and then customize the format of how it is displayed in the interface. Hopefully, from this part we are able to interact with the data and get our application to test how to plan ahead. In the next section we shall cover how to map up all of these together to create something meaningful and finally understand the Django development process. Thank you for reading, if you have any questions or improvements to suggest, please let me know in the comments. I'll be grateful if you you provide a feedback. Happy Coding :) https://www.meetgor.com/posts/django-basics-models Introduction We have seen the basics of Django templating in the previous parts of the series. Now, we can move on to the more backend stuff in Django which dea Tue, 04 Jan 2022 00:00:00 UTC ## Introduction We have seen the basics of Django templating in the previous parts of the series. Now, we can move on to the more backend stuff in Django which deals with the Databases, queries, admin section, and so on. In this particular part, we'll cover the fundamental part of any application in Django i.e the `Model`. We'll understand what the model is, how to structure one, how to create relationships and add constraints on the fields, etc. ## What ate Models? A model is a Django-way(Pythonic) to structure a database for a given application. It is technically a class that can act as a table in a database generally and inside of the class, the properties of it act as the attributes of that database. It's that simple. Just a blueprint to create a table in a database, don't worry about what and where is our database. We will explore the database and its configuration in the next part. By creating a model, you don't have to write all the basic SQL queries like ```sql CREATE TABLE NAME( attrb1_name type, attrb2_name type, . . . ); ``` If your application is quite big or is complex in terms of the relations among the entities, writing SQL queries manually is a daunting task and also quite repetitive at times. So Django handles all the SQL crap out of the way for the programmer. So Models are just a Pythonic way to create a table for the project/application's database. ## How to create a Model? Creating a model for an application is as easy as creating a class in python. But hey! It's more than that as there are other questions to address while designing the class. You need to design the database before defining the fields in the model. OK, we'll it's not straightforward as it seems to but still for creating simple and dummy projects to start with. You can use certain tools like [lucidchart](https://www.lucidchart.com/pages/database-diagram/database-design-tool), [dbdiagrams.io](https://dbdiagram.io/home), and other tools you are comfortable with. It's important to visualize the database schema or the structure of the application before tinkering with the actual database inside the project. Let's not go too crazy and design a simple model to understand the process. Here's a basic model for a Blog: ```python #from django.db import models from django.contrib.auth.models import User class Article(models.Model): title = models.CharField(max_length=255) post = models.TextField() author = models.ForeignKey(User, on_delete=models.CASCADE, related_name='Article') created = models.DateTimeField(auto_now_add=True) updated = models.DateTimeField(auto_now=True) ``` Ignore the `from django.db import models` as it is already in the file created by Django. If not, please uncomment the line and that should be good to go. This is a basic model you might wanna play with but don't dump it anywhere. We define or create our models in the application inside the project. Inside the application there is already a file called `models.py` just **append** the above code into it. The application can be any application which makes the most sense to you or better create a app if not already created and name it as `article` or `post` or anything you like. If you are familiar with Python OOP(object-oriented programming), we have basically inherited the `models.Model` class from the `django.db` module into our model. If you want more such examples, let's see more such models : An E-Mail application core model. Attributes like `sender`, `subject` of the mail, `body` of the mail, `recipients_list` i.e. the `To:` section in a mail system and the `attachment_file` for a file attachment to a mail if any. ```python #from django.db import models from user import EmailUser class EMail(models.Model): sender = models.EmailField(max_length = 255) subject = models.CharField(max_length = 78) body = models.CharField(max_length = 40000) recipients_list = models.ManyToManyField(EmailUser, related_name = 'mail_list') attachment_file = models.FileField(blank=True) ``` A sample model for a note-taking app, consisting of a Note and a Book. A book might be a collection of multiple notes i.e. a single book can have multiple notes so we are using a `ManyToManyField`, what is that? We'll see that shortly. ```python from django.db import models from user.models import User class Notes(models.Model): author = models.ForeignKey(User, on_delete=models.CASCADE) title = models.CharField(max_length = 1024) content = models.Textfield() created = models.DateTimeField(auto_now_add = True) modified = models.DateTimeField(auto_now = True) book = models.ManyToManyField(Book, related_name = 'book') class Book(): name = models.CharField(max_length = 1024) ``` These are just dummies and are not recommended to use anywhere especially in a serious project. So, we have seen a model, but what are these fields and the constraints like `on_delete`, `max_length`, and others in the upcoming section on fields. ## Fields in Django Fields are technically the attributes of the class which here is the model, but they are further treated as a attribute in a table of a database. So the model becomes a list of attributes which will be then parsed into an actual database. By creating attributes inside a class we are defining the structure for a table. We have several types of fields defined already by django for the ease of validating and making a constrained setup for the database schema. Let's look at some of the types of fields in Django Models. ### Types of Fields Django has a lot of fields defined in the models class. If you want to go through all the fields, you read through the django docs [field references](https://docs.djangoproject.com/en/4.0/ref/models/fields/#model-field-types). We can access the fields from the `models` module like `name = models.CharField(max_length=10)`, this is a example of defining a attributes `name` which is a CharField. We can set the max_length which acts a constraint to the attribute as we do not want the name field to be greater than 10 and hence parsing the parameter `max_length` to 10. We have other field types like: - `IntegerField` -> for an integer value. - `TextField` -> for long input of text (like text area in html). - `EmailField` -> for an single valid email field. - `DateField` -> for inputting in a date format. - `URLField` -> for input a URL field. - `BooleanField` -> for a boolean value input. And there are other fields as well which can be used as per requirements. We also have some other fields which are not directly fields so to speak but are kind of relationship defining fields like: - `ForeignKey` -> Define a many-to-one relationship to another model/class. - `ManyToManyField` -> define a many-to-many relationship to another model/class. - `OneToOneField` -> define a one to one relationship between different tables/model/class. So, that's about the field types for just a feel of how to structure or design a database table using a model with some types of attributes. We also need to talk about constraints which needs to added to the fields inside the models. ### Field Options/Arguments We can add constraints and pass arguments to the fields in the models. We can add arguments like `null`, `blank`, `defualt`, `choices`, etc. - `null=True/False` -> Set a check for the entry in the table as not null in the database. - `blank=True/False` -> Set a check for the input validation to empty or not. - `unique=True/False` -> Set a constraint to make the entry unique throughout the table. - `defualt=anyvalue` -> Set a default value for the field. - `choices=list` -> Set a list of defined choices to select in the field (a list of two valued tuple). We also have another constraint specific to the fields like `max_length` for `CharField`, `on_delete` for ForeignKey which can be used as a controller for the model when the related model is deleted, `verbose_name` to set a different name for referencing the entry in the table/model from the admin section compared to the default name of the model, `verbose_name_plural` similar to the `verbose_name` but for referencing the entire table/model. Also `auto_now_add` and `auto_now` for `DateTimeField` so as to set the current date-time by default. More options and arguments that can be passed to the fields in models are given in the django docs [field options](https://docs.djangoproject.com/en/4.0/topics/db/models/#field-options) These are some of the options or arguments that we can or need to pass to the fields to set up a constrained schema for our database. ### Meta class Meta class is a nested class inside the model class which is most of the times used for ordering the entries(objects) in the table, managing permissions for accessing the model, add constraints to the models related to the attributes/fields inside it, etc. You can read about the functionalities of the Meta class in the [documentation](https://docs.djangoproject.com/en/4.0/ref/models/options/). ## Model methods As a class can have functions, so does a model as it is a Python class after all. We can create kind of a helper methods/functions inside the model. The model class provides a helpful `__str__()` function which is used to rename an object from the database. We also have other predefined helper functions like `get_absolute_url` that generates the URL and returns it for further redirection or rendering. Also, you can define the custom functions that can be used as to help the attributes inside the model class. ## Django ORM Django has an Object Relational Mapper is the core concept in Django or the component in Django that allows us to interact with the database without the programmer writing SQL/DB queries. It is like a Pythonic way to write and execute sql queries, it basically abstracts away the layer to manually write SQL queries. We'll explore the details of how the ORM works under the hood but it's really interesting and fascinating for a Beginner to make web applications without learning SQL(not recommended though personally). For now, its just magical to see Django handling the DB operations for you. You can get the references for learning about the Queryset in ORM from the [docs](https://docs.djangoproject.com/en/4.0/ref/models/querysets/) ## Example Model Let us set up a model from what we have learned so far. We'll create a model for a Blog Post again but with more robust fields and structure. ```python #from django.db import models from django.contrib.auth.models import User class Article(models.Model): options = ( ('draft', 'Draft'), ('published', 'Published'), ) title = models.CharField(max_length=255, unique=True) slug = models.SlugField(max_length=255, unique_for_date='publish') post = models.TextField() author = models.ForeignKey(User, on_delete=models.CASCADE, related_name='Posts') created = models.DateTimeField(auto_now_add=True) updated = models.DateTimeField(auto_now=True) status = models.CharField(max_length=16, choices=option, default='draft') def __str__() return self.title class Meta: ordering = ('-publish',) ``` We can see in the above model that we have defined the Meta class which is optional and is generally written to modify how to entries inside the table appear or order with other functionalities as well. We have also added the choices option in the status field which has two choices `Draft` and `Publish` one which is seen by the django interface and the other to the end-users. We have also added certain fields like slug that will create the URL for the blog post, also certain options like `unique` has been set to restrict duplicate entries being posted to the database. The `related_name` in the `ForeignKey` refers to the name given to the relation from the Article model to the User model in this case. So, we can see that Django allows us to structure the schema of a database. Though nothing is seen as an end result, when we configure and migrate the model to our database we will see the results of the hard work spent in creating and designing the model. ## Database Specific fields By this time, you will have gotten a feel of what a database might be. Most of the projects are designed around SQL databases but No-SQL databases and others are also used in cases which suite them the most. We have tools to manage this database in SQL we call it the Database Management System (DBMS). It's just a tool to manage data, but there is not just a single Database management tool out there, there are gazillions and bazillions of them. Most popular include `MySQL`, `PostgreSQL`, `SQLite`, `Oracle`, `Microsoft Access`, `Maria DB`, and tons of others. Well, these different DBMS tools are almost similar with a few hiccups here and there. So, different Database tools might have different fields they provide. For Example, in Database `PostgreSQL` provides the ListField which `SQLite` doesn't that can be the decision to be taken before creating any project. There might be some fields that some DBMS provide and other doesn't. ## Conclusion We understood the basics of creating a model. We didn't touch on the database yet but the next part is all about configuration and migration so we'll get hands-on with the databases. We covered how to structure our database, how to write fields in the model, add constraints and logic to them and explore the terminologies in Django like ORM, Database Types, etc. Thank you for reading the article, if you have any feedback kindly let me know, and until then Happy Coding :) https://www.meetgor.com/posts/django-basics-static-files Introduction After creating templates, it should be rather tempting to add some styles and logic to them. Well yes, we'll see how to add static files in a web a Mon, 27 Dec 2021 00:00:00 UTC ## Introduction After creating templates, it should be rather tempting to add some styles and logic to them. Well yes, we'll see how to add static files in a web application using django. Static files are not only CSS, but also media/images and Javascript files as well. In this part of the series, we'll cover the basics of working with static files in django including the configuration, rendering and storing of the static files. ## What are Static Files? Static files as the name suggests are the files that don't change, your style sheets(css/scss) are not gonna change for every request from the client side, though the template might be dynamic. Also your logo, images in the design will not change unless you re-design it XD So these are the static files that needs to be rendered along with the templates. We have basically 3 types of static files, CSS, Javascript files and media files/static templates,etc. They are all rendered in the same way but as per their conventions and usage. You can learn about the theoretical information on [static files](https://docs.djangoproject.com/en/4.0/howto/static-files/) from the django documentation. ## How to configure Static Files Firstly you can create a folder for all the static files in the root folder. Usually the convention is `static` as the name of the folder. So, if you have created the template folder in the root directory, similar to that static folder can be created in that path. Next after creating the static folder in the project root folder, we need to configure the `settings.py` file to actually tell Django web server to look for all our static files in that folder. To do that, go to the `settings.py` file, now by this time you would have known where the `settings.py` file is (inside the project-named folder). Add the following at the end of the `settings.py` file. ```python # import os # STATIC_URL = '/static/' STATICFILES_DIRS = ( os.path.join(BASE_DIR, "static/"), ) ``` Ignore the `import os` if you already have imported and the `STATIC_URL` if already there in the file. The `STATICFILES_DIRS` is the configuration that we tell the django environment to look for all our static files in the base/root directory of the project where the `static/` folder is. The `os.path.join()` actually gets the path of the directory in our operating system to the folder specified in the case of our project the `BASE_DIR` is the path of the project and we add in the static folder to actually the project path. The final piece and the crucial one is the `"static/"` path, this can be other location where you have created your static folder within the project. That's it! Yes, it's that simple. We can now create static files and render them in our templates. ## Creating and Storing Static files Now this part is customizable and it depends on your preference, how you want to organize the static folder. The convention that I follow is creating separate folders namely for `css`, `js` and `assets`(or `img`) mostly. And inside of this folders you can store the respective static files. This also creates the project more scalable in terms of it's maintenance. ``` static\ |__css\ |__js\ |__assets\ ``` Let's create a static file and an image to demonstrate the concept of static files in django. - css/style.css ```css body { background-color:#1d1dff; color:white; } h1 { text-align:center font-family: monospace; } p { color:#ff6600; font-weight:500; } ul { list-style-type:square; } ``` - assets/tbicon.png Demo Image (that's my blog icon)  ## Rendering Static Files from Templates So, after configuring and creating the static files, we now can inject them into our templates. If you try to do the traditional way i.e. linking stylesheets/images/script files with HTML, it just won't work as you expect to and there's no point in using traditional way while creating a web application with a framework. So, there is a framework specific way to do things which make it easier and efficient for the project. To render any static file, we need to load the static tag which allows us to embed links for the static files into the templates. This means if the static files are not loaded directly instead in production(deploying our application) the static files are stored in a folder `STATIC_ROOT` which the server then loads, we'll see how that internally works when we get to deployment techniques for Django project. To load the static files from our configuration, we can simpy include the tag on top of the template. ``` {% load static %} ``` The above templating tag will load the `static` tag which allows us to embed the links to the static files as explained earlier. Now, we can actually access any file with the static folder in our templates with a particular syntax as below: ```html <link rel="stylesheet" href="#"css/style.css' %}"> ``` Its just a example how to load the file, we are calling the static tag which we have loaded in previously and from there we are referencing the css file. The compact syntax would be : `{% static 'path-to-file' %}` **NOTE: The path to the static file is relative from the Static folder, i.e. enter the path of the file considering the static folder as the base directory.** ### Demonstration of the static file Let's render the static file which we created earlier i.e. the css file and the image into a template. Assuming you have a app called `post` in your django project, you can render static files as below: # templates/home.html ```html <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>Django Blog</title> {% load static %} <link rel="stylesheet" href="#"css/style.css' %}"> </head> <body> <h1>Hello, World!</h1> {% block body %} <p>This is not going to get inherited </p> {% endblock %} <p>This will be inherited</p> </body> </html> ``` We are loading the static tag and then loading the css file using the tag syntax as explained above. # static/css/style.css ```css body { background-color:#1d1dff; color:white; } h1 { text-align:center font-family: monospace; } p { color:#ff6600; font-weight:500; } ul { list-style-type:square; } ``` This is the static file,`style.css` stored inside the css folder of the static folder. This contains basic (very lame) CSS styling as we can understand. # post/views.py ```python from django.shortcuts import render def home(request): return render(request, 'home.html') ``` The `views.py` file has the function that renders the template `home.html` from the templates folder inside the application specific folder. # post/urls.py ```python from django.urls import path from post import views urlpatterns = [ path('', views.home, name="home"), ] ``` This is the application level configuration for the url routes to the views linking the views(functions) from the `views.py` file. The url in this file(code-snippet) is linking the root url('') to the home view in the `views.py` file. # Blog/urls.py ```python from django.contrib import admin from django.urls import path, include from django.views.generic import TemplateView urlpatterns = [ path('admin/', admin.site.urls), path('', include('post.urls')), ] ``` The urls file in the project folder is the core configuration for project level url routes to individual applications within the project. Append the following if your templates and static files are not configured properly. # Blog/settings.py ```python import os TEMPLATES = [ { 'BACKEND': 'django.template.backends.django.DjangoTemplates', 'DIRS': [os.path.join(BASE_DIR, 'templates'),], 'APP_DIRS': True, 'OPTIONS': { 'context_processors': [ 'django.template.context_processors.debug', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.contrib.messages.context_processors.messages', ], }, }, ] STATIC_URL = '/static/' STATICFILES_DIRS = ( os.path.join(BASE_DIR, "static/"), ) ``` SO, the result of the above code is as simple template as shown in the picture below:  This will also work if you do it with traditional HTML syntax, but I'd explained why it's not recommended to do it while using frameworks. Let's see how static files are rendered in inherited templates. We'll tinker with the `for.html` template created in the [previous part](https://mr-destructive.github.io/techstructive-blog/django/python/web-development/2021/12/14/Django-Basics-P5.html). # template/for.html ```django {% extends 'home.html' %} {% load static %} {% block body %} <img src="#"assets/tbicon.png' %}" height="50px" width="50px" /> <ul> {% for sport in sport_list %} <li>{{ sport }}</li> {% endfor %} </ul> {% endblock %} ``` We will have re-load the static tag for each template only if we need to include a new static file in the template. So we use the `{% load static %}` again as we are loading the static file (image) in this template. # post/views.py ```python from django.shortcuts import render def for_demo(request): sports = ('football', 'cricket', 'volleyball', 'hockey', 'basketball') return render(request, 'for.html', {'sport_list': sports}) def home(request): return render(request, 'home.html') ``` # post/urls.py ```python from django.urls import path from post import views urlpatterns = [ path('', views.home, name="home"), path('for/', views.for_demo, name="fordemo"), ] ``` So, that's the url and view map created, we can now be able to see the result in the `127.0.0.1:8000/for/` url to see the below result:  The list style has been changed and thus we can see that the CSS from the parent template is also being inherited. Here is the django project structure which I have created with this series so far:  So that has been it for the Static files in Django. Though there are lot of depth for rendering and loading the static files, we'll explore as we get our grasp in the django and web development terminologies. ## Conclusion So, from this article, we were able to configure and render static files like CSS/Images and optionally Javascript into the Django application. We covered from ground how to configure, load and structure the folder for storing all the static files at the project level. Hope you found it helpful and if you have any queries please let me know. We'll start with the databases probably from the next part in the Django Basics Series. Until then have a great week and as always Happy Coding :) https://www.meetgor.com/posts/django-basics-templates Introduction After learning the basics of views and URLs, we can now move on to the next concept i.e. Templates. In Django, Templates are quite an important com Tue, 14 Dec 2021 00:00:00 UTC ## Introduction After learning the basics of views and URLs, we can now move on to the next concept i.e. Templates. In Django, Templates are quite an important component for the application as it acts as the `frontend` for the web application. With the help of templates and some features provided by Django, it becomes very intuitive and simple to make dynamic web content. In this part, we understand what are templates and what is the way to render them. ## What are Templates Templates are simply a `html` document or kind of a wireframe for content to be displayed for the web app. Templates allow us to render some more relevant pieces of data rather than simple text HTTP responses as we did earlier. We can even re-use certain components of a template in other using the Django Templating Language (more on this later). So, using HTML templates, we can write a complete Webpage. If you are unfamiliar with HTML, you can check out the basics of HTML with this [playlist](https://www.youtube.com/playlist?list=PL081AC329706B2953). Even If you are not familiar with HTML, this tutorial might be quite basic and not overwhelm you with all the tags. ## Creating Templates To create a Template, we can write a simple HTML document like the below: Create a folder `templates` in the base folder, inside the templates folder, create a file `index.html` **templates\index.html** ```html <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>Django Blog</title> </head> <body> <h1>Hello, World!</h1> </body> </html> ``` This is a simple HTML template, with the `<h1>` tags. As Django is a framework, there is a standard for storing all the templates for the project and application. There are a couple of standard of options: - One of which is creating a `templates` folder in the root folder as discussed earlier, also we need to modify the `project_name/settings.py` file. Inside the `settings.py` file, we need to locate the `TEMPLATES` section and modify as below: ```python import os TEMPLATES = [ { 'BACKEND': 'django.template.backends.django.DjangoTemplates', 'DIRS': [os.path.join(BASE_DIR, 'templates'),], 'APP_DIRS': True, 'OPTIONS': { 'context_processors': [ 'django.template.context_processors.debug', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.contrib.messages.context_processors.messages', ], }, }, ] ``` In this snippet, we have changed the `DIRS` option to search the templates in the folder `templates` in the root directory. - The other standard is creating a templates folder in each application. We can create the templates folder in each application instead of a single folder. ## Rendering Templates After creating a template and making the required settings to make sure Django is able to pick up those templates, we need to work with views and URLs to actually render those templates. There are a couple of ways to render templates in Django and some of them are discussed below: ### Using TemplateView [TemplateView](https://docs.djangoproject.com/en/3.2/ref/class-based-views/base/#django.views.generic.base.TemplateView) is a class which is comes with `django.views.generic` library. This class allows us to render a template by providing in the name of the template, arguments or variables to be parsed, and so on. The simplest way to render a template is by the following way: ```python from django.contrib import admin from django.urls import path, include from django.views.generic import TemplateView urlpatterns = [ path('admin/', admin.site.urls), path('', TemplateView.as_view(template_name="index.html"), name="index"), ] ``` We need to import the `TemplateView` from the `django.core.generic` so as to use the class for rendering the template. The `TemplateView` class takes in a couple of arguments, we'll use the `template_name` as an argument that takes in the name of the template. Here, we use the `index.html` as the template which we created earlier. We don't need to specify the entire path to the template as we make modifications in the `settings.py` file to pick the template from the mentioned directory. We use `as_view` function to load the class as a function/view. Activate the virtual environment for the proper functioning of the project. After activating the virtual environment we can run the server as follows: ```terminal python manage.py runserver ``` We can now see the following output and thus, we are now rendering a simple HTML template in Django.  ### Using render We can also use the [render function](https://docs.djangoproject.com/en/4.0/topics/http/shortcuts/#render) from `django.shortcuts` to simply render a template. But we will create a Python function or a View to actually render the template. So, we'll create a View-URL map as we created in the [previous part](https://mr-destructive.github.io/techstructive-blog/django/python/web-development/2021/11/30/Django-Basics-P4.html). Firstly, let's create a view function in the `post/views.py` file, more generally (`app_name/views.py` file). Firstly, we need to import the render function from `django.shortcuts` and then return the function call of render. ```python from django.shortcuts import render def home(request): return render(request,'index.html') ``` And in the URLs, we'll create a different pattern like for e.g. 'home/' ```python from django.urls import path from post import views urlpatterns=[ path('home/',views.home,name="home"), ] ``` So, after creating the View-URL map and making sure the URL of the app is loaded in the project URLs, we can see the result as a simple HTML template.  ## Django Templating Language The [Django Templating Language](https://docs.djangoproject.com/en/3.2/ref/templates/language/) is Django's way of making templates more dynamic and easy to write dynamic web applications. We'll take a brief look at what we can do with this type of Templating Language in Django. ### Variables This is the most common use case for the Django Templating Language/Engine as we can use the [variables](https://docs.djangoproject.com/en/3.2/ref/templates/language/#variables) from the Backend and inject it in the template. We can parse the variable into the template by the syntax : `{{ variable_name " }}}}` To show its use cases, we can declare a variable in a view and then parse it in the Template. Though it is not dynamic right now we can later on fetch values from the database and store them in the form of variables in our views. **templates/home.html** ```html <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>Django Blog</title> </head> <body> <h1>Hello, {{ name }}</h1> </body> </html> ``` **post/views.py** ```python from django.shortcuts import render def variable_demo(request): name = "Kevin" return render(request, 'home.html', {'name':name}) #The name can be anything, like a database query object, form detail, etc ``` As we can see the variable in views is passed as a dictionary in python. The reference key along with a value of the variable as the name of the variable. We will use the key in the templates to parse the value of the variable. **post/urls.py** ```python from django.urls import path from post import views urlpatterns=[ path('home/',views.home,name="home"), path('vardemo/',views.variable_demo, name="var"), ] ```  As we can see, we were able to load the variable into the template using the Django Templating Engine. ### Conditional statement We can even use the conditional statement in the Template using a very simple syntax. We can use `{% if condition" }} %}` to use certain special kinds of blocks in the Template. We need to end those blocks as well using the syntax `{% endif %}`, here `if` can be other blocks which we'll explore ahead. To create a basic if condition in the template, we can understand with the following example. **app_name/views.py** ```python from django.shortcuts import render from random import randint def if_demo(request): number = randint(1,10) return render(request, 'if_else.html', {'num':number}) ``` Here, we have used the key name as `num` indicating we can give different names to the key which needs to be used in the template to render the values. **app_name/urls.py** ```python from django.urls import path from post import views urlpatterns = [ path('if/', views.if_demo, name="ifdemo"), ] ``` **templates/if_else.html** ```html <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>Django Blog</title> </head> <body> {{ num }} {% if num > 5 %} <h2>It's Greater then 5</h2> {% elif num == 5 %} <h2>It's five!</h2> {% else %} <h2>It's less than 5</h2> {% endif %} </body> </html> ```  So, as we can see that, we can use the if-else conditions in the template and that is already powerful. This can be a bit messy as to handle mathematical operations or conditions into a single condition. This can really be used for really large datasets that can be shimmed down to really less coding and also improve readability. ### For loop Now, the most crucial component of the Django templating language is the loops. We can actually iterate over objects/lists in the template. This becomes a huge concept for actually making a dynamic web application. We n\might want to iterate over all the entries in a database, or any other form of data which can make the app a lot dynamic and feel real-time. The syntax of for loop is almost similar to the if-else condition. We just replace the condition with the iterator and the list/object from the view context. `{% for i in list %}`, also end the for loop like `{% endfor %}`. **app_name/views.py** ```python from django.shortcuts import render def for_demo(request): sports = ('football', 'cricket', 'volleyball', 'hockey', 'basketball') return render(request, 'for.html', {'sport_list': sports}) ``` We have created a simple Python list called `sports` and we parse them to the template using a dictionary object, `sport_list` as the key for storing the value of the `sports` list. **app_name/urls.py** ```python from django.urls import path from post import views urlpatterns = [ path('for/', views.for_demo, name="fordemo"), ] ``` **templates/for.html** ```html <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>Django Blog</title> </head> <body> <ul> {% for sport in sport_list %} <li>{{ sport }}</li> {% endfor %} </ul> </body> </html> ```  We have used simple for loop in Pythonic syntax, we use an iterator in this case, `sport` acts as an iterator. We use this to store values one by one from the list `sport_list` which was earlier passed in the views as a key in the dictionary. Hence, this is quite scalable and used to fetch the objects/entries in the database and hence making it a lot easier to make a dynamic web application faster. ## Template Inheritance So, far we have seen that we need to create the base template again and again like all the basic HTML elements, title, and all the basic structure. But what if, we can reuse a specific template in another and extend the functionality of that template into a new one. This avoids the redundancy of writing the entire basic template or the layout of a web app over and over again. To do that, Django has the Template inheritance. We can use a template as its basic layout or a specific component in the web application. Again, similar to the for, if-else blocks the syntax for inheriting a template is quite similar. Take, for example, the home.html which consisted of only a `<h1>` tag in it. We can use this kind of template in other templates to really make it the home page. For that, we first need to enclose the template in a `block`, which is what allows us to use it in other templates. To create a `block`, we simply need to write the following syntax before the component which we do not want in other templates: **templates/home.html** ```html <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>Django Blog</title> </head> <body> <h1>Hello, World!</h1> {% block body %} <p>This is not going to get inherited </p> {% endblock %} <p>This will be inherited</p> </body> </html> ``` In this we have used the `blocks` with a name like `body` as `{% block body %}` this can be anything you like. We end the block with the similar syntax as the for/if blocks as `{% endblock %}`. Anything in between the blocks i.e `block block_name` and `endblock` is not inherited i.e it is unique to this template. We will see how we can use this template in other templates. We will actually extend this template and use the blocks to render the content in the template. **templates/if_else.html** ```html {% extends 'home.html' %} {% block body %} {{ num }} {% if num > 5 %} <h2>It's Greater then 5</h2> {% elif num == 5 %} <h2>It's five!</h2> {% else %} <h2>It's less than 5</h2> {% endif %} {% endblock %} ``` So, we first say to Django to extend the `home` template i.e. the Django will load the blocks from this template only, remember it will just load and not use the blocks until we explicitly tell it to. To use the blocks or kind of plug in the template content in the `if_else.html` or any other template, we need to again call the `blocks`. Here, we need to write the content inside the `blocks` to properly parse the blocks as this is an HTML template. The order of opening and closing elements do matter. So, when we say `endblock` the last part of the base template is loaded i.e. the closing `body` and `html` tags. This is like plugging the template as it is before and after the block body. **app_name/views.py** ```python from django.shortcuts import render def home(request): return render(request, 'home.html') from random import randint def if_demo(request): number = randint(1,10) return render(request, 'if_else.html', {'num':number}) ``` **app_name/urls.py** ```python from django.urls import path from post import views urlpatterns = [ path('', views.home, name="home"), path('if/', views.if_demo, name="ifdemo"), ] ```    The above gif illustrates the example in a neat way. The block is loaded from the given template as the extended template and hence it plugs the block into the frame of the template. ## Conclusion So, from this part of the series, we were able to understand the concept of Templates in Django, we were able to use variables, loops, conditional statements, and template inheriting in a Django application. In the next part, we'll try to touch up with the static files and see how to properly structure and configure them. Thank you for reading, if you didn't understand any of the examples, please let me know, I'll be happy to share the code. Happy Coding :) https://www.meetgor.com/posts/django-basics-views-urls Introduction After getting familiar with the folder structure of the Django framework, we'll create our first view in an app. The basics of creating and mapping Tue, 30 Nov 2021 00:00:00 UTC ## Introduction After getting familiar with the folder structure of the Django framework, we'll create our first view in an app. The basics of creating and mapping a view with a URL will be cleared by the end of this part. ## Creating Views > Views are the functions written in python as a logic control unit of the webserver To create a view or typically-like function, we need to write a function in the `views.py` file inside of the application folder. The function name can be anything but should be a sensible name as far as its usability is concerned. Let's take a basic example of sending an HTTP response of "Hello World". #### project_name/app_name/views.py ```python from django.http import HttpResponse def index(request): return HttpResponse("Hello World") ``` Yes, we are simply returning an HTTP Response right now, but rendering Templates/HTML Documents is quite similar and easy to grasp in Django. So, this is a view or a piece of logic but there is a piece missing in this. Where should this function be used? Of course a URL i.e a path to a web server. We'll see how to map the views to an URL in Django in the next section ## Mapping the Views to a URL We need to first create a `urls.py` file in the application folder to create a map of the URL to be mapped with the view. After creating the file in the same app folder as the `views.py`, import the function in the view into the file. #### project_name/app_name/urls.py ```python from .views import index from django.urls import path urlpatterns = [ path('', index, name="index"), ] ``` The path can be anything you like but for simplicity, we'll keep it blank('') for now. Now, you have the path for your view to work but it's not linked to the main project. We need to link the app urls to the project urls. To link the urls of your app to the main project folder, you need to just add a single line of code in the `urls.py` file of the project folder. In projectname folder -> urls.py #### project_name/urls.py ```python from django.contrib import admin from django.urls import path, include urlpatterns = [ path('admin/', admin.site.urls), path('', include('post.urls')), ] ``` You need to add the line `path('', include('post.urls')),` and also import the `include` function from `django.urls`. This additional statement includes the urls or all the `urlpatterns` in the `post` app from the `urls.py` file into the project's url-routes. Here, the URL path can be anything like `'home/'`, `'about/'`, `'posts/'`, etc. but since we are just understanding the basics, we'll keep it `''` i.e. the root URL. You can also see that there is another route in our project `'admin/'` which is a path to the admin section. We'll explore this path and the entire Admin Section in some other part of this series. Now if you start the server and visit the default URL i.e `http://127.0.0.1:8000`, you will see a simple HTTP message `Hello World`.  ## Breaking the `path` function in urlpatterns The path function in the urlpatterns takes in at least 2 parameters, i.e. the URL pattern and the view of any other function that can be related to the webserver. ``` path( ' ', view, name ) ^ ^ ^ | | | | | url_name | function_name url_path ``` ### URL path The URL Path is the pattern or literally the path which you use in the Browser's search bar. This can be static i.e. some hard-coded text like `home/`, `user/`, `post/home/`, etc. and we can also have dynamic URLs like `post/<pk:id>/`, `user/<str:name>/`, etc. here the characters `<pk:id>` and `<str:name>` will be replaced by the actual id(integer/primary key) or the name(String) itself. This is used in an actual web application, where there might be a user profile that needs the unique user-id to render it specifically for that user. The User-Profile is just an example, it can anything like posts, emails, products, any form of a content-driven application. ### View The view or the function is the name of the function that will be attached to that URL path. That means once the user visits that URL, the function is literally called. **View is just a fancy word for a function(or any logic basically).** There is a lot to be covered when it comes to `View` as there are a lot of ways to create it, there are two types of views, how to use them for various use-cases that can be learned along the way because it is a topic where the crust of Django exists. We'll learn to create different implementations and structure our views, for time-being just consider them as the unit where every operation on the web can be performed. We can create other standalone functions in python to work with the views to make it a bit structured and readable. ### URL Name This is an optional parameter to the path function as we do not mandatorily need to give the URL map a name. This can be really useful in multi-page application websites where you need to link one page to another and that becomes a lot easier with the URL name. We do not need this right now, we'll touch it when we'll see the Django Templating Language. ## Example Views Let's create some examples to understand the working of Views and URLs. We'll create a dynamic URL and integrate the Python module in the views to get familiarized with the concept. ### Dynamic URLs We can use the dynamic URLs or placeholder variables to render out the content dynamically. Let's create another set of View and URL map. #### project_name/app_name/views.py ```python def greet(request, name): return HttpResponse("Welcome, " + name) ``` This view or function takes an additional argument called `name` and in response, it just says `Welcome, name` where the name can be any string. Now after creating the view, we need to map the view to a URL pattern, We'll add a path for this greet function. #### project_name/app_name/urls.py ```python path('greet/<str:name>/', greet, name="greet"), ``` You can see how we have created the url-pattern here. The greet part is static but the `<str:name>` is a variable or just a URL parameter to be passed to the view as the value of the variable `name`. We have also given the URL map a name called greet, just for demonstration of its creation. You'll get an error, 100% if you are blindly following me! Didn't you forget something? Import the greet function from the views like so: ```python from .views import index, greet ``` So, after we visit the URL `https://127.0.0.1:8000/greet/harry`, you should see a response `Welcome, harry` as simple as that.  Now, how is this working? We see the view first. The function takes two parameters one is most common the request which stores the meta-data about the request, the other parameter is the name that we will be use to respond to the server dynamically. The name variable is used in the string with the HttpResponse function to return a simple string. Then, in the URLs, we need to find a way to pass the variable name to the view, for that we use the `<string:name>` which is like a URL parameter to the view. The path function automatically parses the name to the appropriate view and hence we call the greet function with the name variable from the URL. ### Using Pythonic things We'll use some Python libraries or functions in the Django App. In this way, we'll see it's nearly no-brainer to use Python functions or libraries in the Django framework as indeed all files which we are working with are Python files. #### project_name/app_name/views.py ```python from random import randint def dice(request): number = randint(1,6) return HttpResponse(f"It's {number}") ``` This view is using the random module, you can pretty much use other web-compatible modules or libraries. We have used the `random.randint` function to generate the pseudo-random number between 1 and 6. We have used the f-string (`f"{variable}"`)styled Response string as int is not compatible with the response concatenation. So this is the logic of our map, now we'll need to link it to a URL-path. #### project_name/app_name/urls.py ```python path('throw/', dice, name="dice"), ``` Also, import the view name from the views as `from .views import dice` also add other views if present. Now if we go to the URL `https://127.0.0.1:8000/throw/`, we shall see a random number in the response. This is how we used Python to make the logic of our view.  So, that was the basics of creating and mapping views and urls. It is the most fundamental of the workflow in Django project development. You need to get familiar with the process of mapping Views and urls before diving into Templates, Models, and other complex stuff. ## Conclusion From this part of the series, we touched upon the basics of views and URLs. The concept of mapping URLs and views might have been much cleared and it will be even gripping after we explore the Template handling and Static files in the next part. If you have any queries or mistakes I might have made please let me know. Thanks for reading and Happy Coding :) https://www.meetgor.com/posts/django-basics-folder-struct Introduction After setting up the development for the Django framework, we will explore the project structure. In this part, we understand the structure along w Wed, 24 Nov 2021 00:00:00 UTC ### Introduction After setting up the development for the Django framework, we will explore the project structure. In this part, we understand the structure along with the various components in the Project as well as individual apps. We will understand the objective of each file and folder in a brief and hopefully by the end of this part, you'll be aware of how the Django project is structured and get a good overview of the flow of development in the Django project. ## Project Structure We will create a Django project from scratch and understand it from the ground up. As in the previous part, I've shown you how to create a project. In this section, we'll create a project `Blog`. TO do that, we'll create a folder called `Blog`, install and set up the virtual environment as discussed and explained in the previous part. After the virtual environment is created and activated, we'll create the project. ```shell django-admin startproject Blog . ``` After this command, if we see the directory structure, it should look something like this:  As we can see there are 6 files and a folder. The base folder is for the configuration at a project level. I have actually not shown the `venv` ( using `-I venv` option on tree command) as it is out of the scope of this series. The `venv` folder contains modules and scripts which are installed in the virtual environment. So, lets break the folder structure down into understandable files.  ### manage.py Our project consists of a `manage.py` file which is to execute several commands at a project level. We do not have to edit any of the contents of this file (never). It is the file that allows us to run the server, apply migrations, create an Admin account, create apps, and do a lot of crucial things with the help of python. So, it's just the command-line utility that helps us interact with the Django project and applications for configurations. ### Project Folder Now, this is the folder where the project-configuration files are located. **The name of the folder is the same as that of the project**. This makes the folder unique and hence creates a standard way to store files in a structured way. The folder is a python package which is indicated by the `__init__.py` file. The purpose of the `__init__.py` file is to tell the Python environment that the current folder is a Python package. The folder consist of several files(5 files): ### settings.py This is a really important file from the project's point of view. This contains certain configurations that can be applied to the rest of the project (or all the apps). In the `settings.py` file, we can do some of the following operations : - List of `applications` that might be pre-installed or user-defined. - Configure the Middleware. - Configure and connect the Database. - Configure Templates/Static/Media files. - Custom Configuration for Time-Zones, Emails, Authentication, CORS, etc. Besides the above-mentioned options, there is a lot of project-specific configurations or application-specific settings as well. Here, you'll have a question, ### WHAT IS AN APPLICATION? An application is a component of a project. There are also Python packages that are made to be used as a Django app that allows reusing the components. But when we are developing the project, we can break a complex process/project into individual apps. For Example, a project of `Blogging Platform` might have an application for `posts`, `users`, `api`, `homepage`, etc. So the project `Blogging Platform` might have separated the components like its API, Homepage, Post, Users, and so on to keep the development independent and well organized. So, we can understand apps as separate components of a large project. We can also understand apps as reusable components, you can use the `posts` app in another project or in a particular app of the same project making it easier and faster to create the project. ### urls.py This is a file for managing the `URL` routes of the project. We'll discuss URLs and Views in their own part in the series. This file basically has a list of URLs that should be paired with a `view` or any other function. In the project folder, the URL patterns mostly link a baseurl to the URL file of the particular application. Don't worry if you can't get some of the terms, you'll clearly understand when we see them in the future parts of this series. ### wsgi.py WSGI or Web Server Gateway Interface is a file that is used to configure the project for production or deployment. This takes care of the server when we deploy into production. It is a Synchronous Web Server i.e. it listens to only one request and responds to that at a time. Some of the common WSGI servers are [Gunicorn](https://gunicorn.org/), [Apache](https://docs.djangoproject.com/en/3.2/howto/deployment/wsgi/modwsgi/), [uWSGI](https://docs.djangoproject.com/en/3.2/howto/deployment/wsgi/uwsgi/), [cherrypy](https://docs.cherrypy.dev/), [Aspen](https://github.com/buchuki/aspen/blob/master/aspen/wsgi.py), etc. ### asgi.py ASGI or Asynchronous Server Gateway Interface is also similar to the WSGI file but it serves as an asynchronous web server. This file handles the requests which might be asynchronous i.e. the web server can respond to multiple requests and respond to them at a time. We can even send tasks to the background using this type of server configuration. Some of the common ASGI servers are [Uvicorn](https://www.uvicorn.org/), [Daphne](https://docs.djangoproject.com/en/3.2/howto/deployment/asgi/daphne/), [Hypercorn](https://docs.djangoproject.com/en/3.2/howto/deployment/asgi/hypercorn/), etc. ## Creating a Django Project Application So, let's create an application to see the structure of the basic app in Django. To create an app, we can use the `startapp` option with the `python manage.py` command followed by the name of the app like: ```shell python manage.py startapp name ``` Here, `name` can be any app name you'd like to give. ## Application Structure After creating an app, let the name be anything it should have a similar structure as :  As we can see there are a couple of things to be discussed here. The main components that we are going to work on within developing the application in the project are: `models.py`, `views.py`, `test.py`. There are other files that we will create manually like the `urls.py`, `serializers.py`, etc. You also need to add the name of the app in quotes in the `INSTALLED_APPS` list in the `settings.py` file. Something like this:  The application files can be summarized as :  Let us look at each of the files and folders in the application: ### models.py As the same suggests, we need to define the model of a database here. The actual structure and the relationship are created with the help of python and Django in this file. This is the place where the crust of the web application might be defined. There are various aspects in creating a model like `Fields`, `Relationship`, `Meta-data`, `methods`, etc. These are defined with the help of python along with the Django Models. In most cases, a model is like a single `table` in an actual database. The file is quite important and interesting as it abstracts away the manual work of typing `SQL` queries to create the database. ### migrations This migrations folder is a way for Django to keep track of the changes in the database. At every `migration` or actual query that runs to create the table or the database structure. There might be multiple steps or iteration of the database, this folder stores those pieces of information. To make an analogy, it is like a `.git` folder but for keeping track of the migrations or changes to the database. ### admin.py This is the file for performing the operations at the admin level. We generally use this file to register the models into the `Admin section` without touching any frontend part. It provides a built-in `CRUD`(Create Read Update Delete) functionality to the model. This is really good for testing up the model manually before putting effort into the frontend part. Other than this, we can customize the admin section with this file. We will see the details in the part of the `Admin section` in this series. ### views.py This is a file, that acts as a controller/server logic for the Django framework. We can define functions and classes as a response to the incoming requests from the server via the `urls.py` file. There are a couple of approaches when it comes to writing the format of the functions like `class-based views`, `function-based views`, and others depending on the type of operation is been done. As said earlier, it is the `V`(View) in the `MVT` architecture in Django Framework. This is the place where we write the logic from the server-side to let's say render HTML pages(templates), query to the database with CRUD operations, return an HTTP response, etc. ### urls.py This is the file in which a list of URL patterns is mapped to the particular view function. This `urls.py` is specific to the app and it might be prefixed with the URL route mentioned in the project folder's `urls.py` file. So, not going much deeper, simply to put it's a map of a particular URL path with a function associated to it which gets triggered(called) when the user visits the URL. ### tests.py This is a file where we can perform automated tests on the application. This might be in integration with models, other applications, project settings, etc. This is a component that Django makes it to have easy and quick unit testing along with the Python modules for advanced testing. It is quite easier to integrate python modules and libraries into almost anything in the Django Project. ### apps.py This is the file for app-level configuration. We can change the default fields, app name, email settings, other module-specific settings that can be used in the models, views, or in another place that can be defined here. ## Other Folders/files Apart from the app folder and the project folder, we may have other folders like the `templates`, `static`, `media`, etc. There are also python package-specific folders for which you may need to create folders. ### Templates There are a couple of standard ways you can set up your Templates folder. Either in the root project or inside individual apps. The choice is yours, however, you feel comfortable. I personally use only one template folder in the root directory, but you can keep it wherever you want, but these two are the standard ones for ease of reading and maintaining the projects. ### Static The Static folder is the folder in which you store your `css`, `javascript`, and `images`(images or media files that are used in the templates). This is a good way to improve the performance as in the production the webserver collects all the static files and stores them in a single place for responding to the requests. The template folder if present in the root folder, has a sub-folder as the application names and inside the `app-name`, we put in all the `.html` or other template files. As similar to the `template` folder, the location can be modified or set as a configuration from the settings.py file. Usually, the static files(`.css`, `js`, etc) are stored in the root folder with app names as subfolders. ### Media The media folder is where you can store the media-specific to the user or the application processed data. For example, we can store the profile pictures of users, email attachments if it's an email application, thumbnails of the posts for a blogging platform, etc. The configuration of the Media folder is quite similar to the Static folder but it has certain additional configurations. We'll look at them in their sections in this series. Phew! That was all the folder structure you need to get started with Django. There might be other folders and files specific for project, application, python modules but it won't be much hard to understand those as well. ## Conclusion From this part, we were able to understand the folder structure of the Django framework. We explored the various files and folders with their use cases and their purpose. So, by reading the above description of the files and folders you might have got a rough idea about the flow of the development cycle in Django. In the next part, we'll start with actually getting hands dirty in the code and making our first view. Thank you for reading and Happy Coding :) https://www.meetgor.com/posts/django-basics-setup Introduction The crucial aspect of starting to learn any framework is the ease to set it up and Django by far is the easiest of the options out there. There is Sat, 20 Nov 2021 00:00:00 UTC ## Introduction The crucial aspect of starting to learn any framework is the ease to set it up and Django by far is the easiest of the options out there. There is just a few lines of code to install django if you already have python installed in your system. In this article, we see how to setup a django project along with a virtual environment. If you already have python and pip installed, you can move on to the [virtual environment setup](#setting-up-virtual-environment-in-python). ## Installing Python and PIP Django is a python based framework so that makes sense to have Python installed along with its package manager to use Django. To install Python, you can visit the official [Python](https://www.python.org/downloads/) website to download any relevant version for your system (recommended 3.7 and above). Mostly the Python installation comes with the option to install `pip`(python's package manager) but if you missed that, that's fine, you can install the [get-pip.py](https://bootstrap.pypa.io/get-pip.py) file into your system and run the below code: ``` python get-pip.py ``` Make sure the include the relative path to the file if you are not in the same folder as the file. So, that should be python setup in your local machine. To check that python was installed correctly, type in `python --version` and `pip --version` to check if they return any version number. IF they do, Congratulations !! You installed Python successfully and if not, don't worry there might be some simple issues that can be googled out and resolved easily. --- Let's move on to the actual setting of the Django project set up. ## Setting up Virtual Environment in python Virtual Environment is a software which isolates the installation of dependencies and libraries for a specific project, making it a clean and safe environment for deployment as well as maintenance. In Python, we have a virtual environment package known as `virtualenv` that does this thing. It is for installing the Python related packages into a isolated folder. So, we can install the `virtualenv` package in python by following the following steps: ### Installing Virtualenv Firstly, install the virtual environment package, it's not mandatory but it keeps things simple and easy for your project in correspondence to the entire OS. So in python, we have a module to create the virtual environment pretty easily, ``` pip install virtualenv ``` You can use `pip3` or `pip -m`, or however you install normal python modules. This just installs the python virtual environment, we need to create a virtual environment in the current folder. ### Creating a virtual environment We need to create the environment so as to give the Python interpreter an indication to consider the current folder as an isolated Python environment. We need to create a virtual environment in the current folder, so for that navigate to the folder where you want to create the project and enter the following command: ``` virtualenv venv ``` Here, `venv` can be anything like `env` just for your understanding and simplicity it's a standard name kept for the same. After this, you will see a folder of the same name i.e. `venv` or any other name you have used. This is the folder where python will keep every installation private to the local folder itself. ### Activating Virtual environment Now, we need to activate the virtual environment, this means that any thing installed in the prompt with the virtualenv activated will be isolated from the entire system and will be installed on in the virtual environment. To activate the environment, we can use the command : #### for Linux/macOS : ``` source venv/bin/activate ``` #### for Windows: ``` venv\Scripts�ctivate ``` After this, your command prompt will have a `(venv)` attached in the beginning. This indicates you are in a virtual environment, things you do here, may it be module installation or any configuration related to python will stay in the local folder itself. ## Installing Django After the virtual environment is set up and activated, you can install Django and get started with it. Django is a python module or package, which can be easily installed using its package manager `pip`. Install Django using pip: ``` pip install django ``` ## Create a Django Project After the installation is completed, you can start a Django project in the current folder from the django package we installed. There are several commands available in the django module which you can execute in the command line that we'll discuss later. For now, we will use the command `startproject` this is one of the [management commands](https://github.com/django/django/tree/main/django/core/management/commands) in Django. The [django-admin](https://docs.djangoproject.com/en/3.2/ref/django-admin/) is a command line utility for doing the administrative tasks related to Django. ``` django-admin startproject myproject ``` Here `myproject` can be your project name. After this, you will see one new folder and one file pop up. Namely, the `<project-name>` folder and `manage.py` file. We don't have to touch the `manage.py` file but we use it in most of the commands to use the Django functionalities, it is quite similar to the `django-admin` command. You can now run your basic server using the command : ``` python manage.py runserver ``` OR You can use `djagno-admin` command, but you need to set certain environment variables and modify the settings.py file as per the project-name. You can use the `django-admin` as the steps given in the django [documentation](https://docs.djangoproject.com/en/3.2/ref/django-admin/#cmdoption-settings). The output of the command `python manage.py runserver` should be visible in the browser at `https://127.0.0.1:8000` as below :  That's it the base django project is installed in your system. To stop the server simply press `Ctrl+C`. Follow the below GIF for a clear understanding of those instructions:  --- ## Quick-Setup-Script You can avoid manually typing the commands once you get the idea of the process in setting up a django project by executing a simple shell script (for Linux/macOS) or a batch script (for Windows). The script looks something like: For Linux/macOS: ```shell #!/usr/bin/env bash mkdir $1 cd $1 pip install virtualenv virtualenv venv source venv/bin/activate pip install django django-admin startproject $1 . clear ``` Save as commands.sh file For Windows: ```batch mkdir %1 cd %1 pip install virtualenv virtualenv env call env\Scripts�ctivate pip install django django-admin startproject %1 . cls ``` save as commands.bat file For further instructions you can checkout the [GitHub repository](https://github.com/Mr-Destructive/django-quick-setup-script) or a detailed [article](https://mr-destructive.github.io/techstructive-blog/django/web-development/python/2021/08/15/Django-Quick-Setup.html) about it. ## Conclusion From this section, we were able to setup the Django project in our local system. In the next part, we will cover the `folder structure` of the Django project. We won't directly go into the code part because that is very easy once you understand the flow of the framework and its internal working. So, thanks for reading and Happy Coding :) https://www.meetgor.com/posts/django-basics-intro Introduction Welcome to Django Basics series, in this series we'll explore the basics of the Django web framework. In this part, we'll understand what this web Tue, 16 Nov 2021 00:00:00 UTC ## Introduction Welcome to Django Basics series, in this series we'll explore the basics of the Django web framework. In this part, we'll understand what this web framework provides and what actually the back-end development consists of. We'll discuss where Django is used and why it is a great choice for beginners as well as experienced developers. ## What is Django? [Django](https://developer.mozilla.org/en-US/docs/Learn/Server-side/Django/Introduction) is a back-end web framework. It is based on python which means you have to write most of the project's code in Python. But Django comes with a lot of boilerplate code and thus it becomes quite quick in the development. Django is an open-source framework, it is maintained by the Django Software Foundation Organization. You can view the source code at [GitHub](https://github.com/django/django). ### BACKEND ? The term `backend` refers to the section or an essential component in Web development, it consists of a `database`, `API`, and the `web server` itself which allows the components to connect together. There might be other components like `load-balancers`, `middleware`, etc. But the core of web applications revolves around **Databases** and **API**. #### Database A database is a technology or tool that lets you store the data which might be used for serving the actual application, that might be a frontend app, standalone API, etc. The data you want to store might be generally the User Accounts, Content of the App, basically any some form of data(there are exceptions here, you can't directly store media files in DB). The Database allows to make content management and the application dynamic and can be personalized. We have certain types of databases like SQL(relational), NO-SQL, Cloud, Network, etc. The tools of these database management are PostgreSQL, MySQL, MongoDB, HarperDB,etc. These tools allow you to manage your database in a convenient way. #### API An API or Application Programming Interface is a way for any frontend app, outside the system to access the database. API allows you to query to the database with GET, POST, DELETE, PUT, etc kinds of operation/requests to the database via the webserver. In API, we have endpoints or (URL routes) at which a particular designated operation can be performed. In APIs, we currently have four primary architectures namely RESTful (quite famous and well established), SOAP, gRPC, and GRAPHQL (new and increasing in popularity). ### Framework? A framework is a tool to do a certain task efficiently and avoid some repetitive patterns by abstracting many layers in developing it. Django is a high-level framework which means it abstracts certain processes in making the application. It is ideal for beginners to get up and running with a professional full-stack web application(though it requires some learning). Django makes the project ideal for experienced as well as beginner web developers. The community and the ecosystem of Python are quite amazing as well as there are a ton of resources to get you through your projects.  The above is a high-level view of how Django project development works, the application might be not only one but several other standalone applications working together to make one project in Django. There is a lot of abstraction in Django like the Middleware, Session Management, Security, etc. This should be a good overview of the development map in Django. Django follows an MVT architecture. Architecture is a standard in developing an application/project for the ease of the workflow and making it an even experience.  The above diagram depicts the architecture in Django, the components in the Django server include the Model, View, and Template. ### Model `Model` refers to the design of the database or a blueprint of the data that is bound with the application in the project. ### View The `View` is the part to control the way the data should be presented or the how response should be given back to a request from the server(client) ### Template The `Template` is the markup or the form of document that is to be rendered on the client-side and these are controlled by the views and parsed with the data from the models. ## Why would you need it? As a developer you would find a need to host your projects on the internet, for that learning and deploying a web server from the ground up might be quite complex and time-consuming, Django solves this problem quite well. Not only it is easy but even scalable at a production level, making it quite a robust choice for anyone. And as a bonus thing, it is based on Python, which makes it even easier to write code for people staying at an abstracted perspective in programming. Python has by far the richest sets of libraries and utilities for any domain, this integration with Django is a deadly combination. #### Batteries included? Django solves many problems by abstracting away many things like managing the database, rendering dynamic templates(HTML), properly structuring and serving static and media files, well-organized project structure, and many other things. You just have to get the actual thing done i.e. the server logic(or how to design the API/Database models). On top of that, Django has a built-in fully fledged Admin section and a User model. An Admin section is where you can manage the project in a better way without touching the code. It also has certain applications/libraries to make the development of APIs, integrating various databases, forms for posting data, support for Bootstrap a lot easier. It's like a `plug and play` kind of thing for the development of web applications. Hence, it is rightly called the `Batteries Included` web framework. ### Key features of Django - Ease in integrating a database - Flawless Django Template Engine - Easy to scale up/down - Python libraries support out of the box - Amazing Documentation / Helpful community - Developing Production-ready projects quickly - Baked in support for testing, APIs, cookies, sessions, etc - Optimized for security, SEO, and DRY(don't repeat yourself) principles ## Applications built with Django Django is used in quite a famous application that you might be using daily. Django along with Python powers the top applications on the internet like: 1. YouTube 2. Instagram 3. Spotify 4. Disqus 5. Dropbox 6. Pinterest 7. National Geographic 8. Mozilla 9. BitBucket 10. Discovery Network You have to say, it is powerful and has firm grounds in the tech industry. It's highly unlikely that Django will be overtaken by another framework at least some years from now. > Django is a tool to build web applications fast and in a scalable and Pythonic way ## What will this series cover? Learning Django from the ground up. We will learn the setup, folder structure, architecture of Django, What are apps, views, URLs, models, serializers, static and template files, and there is a ton of stuff to be covered. ### Resources to learn Django - [Django Official Docs](https://www.djangoproject.com/start/) - [Very Academy - Django Playlist](https://www.youtube.com/c/veryacademy/playlists?view=50&sort=dd&shelf_id=2) - [Codemy.com - Django](https://www.youtube.com/watch?v=HHx3tTQWUx0&list=PLCC34OHNcOtqW9BJmgQPPzUpJ8hl49AGy) - [Corey Schafer](https://www.youtube.com/watch?v=UmljXZIypDc&list=PL-osiE80TeTtoQCKZ03TU5fNfx2UY6U4p) - [Telusko](https://www.youtube.com/watch?v=SIyxjRJ8VNY&list=PLsyeobzWxl7r2ukVgTqIQcl-1T0C2mzau) ## Conclusion From this article, we were able to understand the Django framework, what is it, and why it should be used on a high level. Further, we explored the web application(backend) components which are targeted by Django for ease of developing applications. We also saw the baseline architecture that Django uses to make projects. In the next section, we'll start the actual coding in Django, firstly how to set up the environment and understanding the folder structure, and so on. So I hoped you enjoyed the article. Thank you for reading. Happy Coding :) https://www.meetgor.com/posts/curl-basics Introduction We all might have used the curl command but might be unaware of it. It's super old and still serves a great purpose. It has been available since 19 Fri, 05 Nov 2021 00:00:00 UTC ## Introduction We all might have used the curl command but might be unaware of it. It's super old and still serves a great purpose. It has been available since 1996 and still is widely used in many embedded technologies, web API testing, CLI applications, etc. In this article, we'll see some basics of using the curl command along with its applications. ## What is the curl command? Curl or cURL command is the utility or tool to access the internet from the command line interface using various protocols. This looks trivial but it can blow up your mind! Most people use this tool for fetching and processing the data from the servers/internet from their terminal without the browser but there is a lot more to it. It is used in various embedded devices for accessing the network in a lightweight and accessible way. Let's see how you can use the curl command from the very basics. ## Why do we need it? Before we talk about how to use the curl command let's talk about why might we need that? There are a lot of reasons and it even depends on the application you are using. You can use curl to test your API, well there are other tools like POSTMAN, Insomnia, etc but for keeping things simple you can quickly get in with curl and test some endpoints. You might require curl for creating some CLI applications that require fetching/posting to an URL over the internet. If you are using the terminal, curl integrates really very well with the shell programming languages like BASH, ZSH, etc So, after making WHY out of the way, let's start with the actual content. ## Structure of curl command **curl or Client URL is a command-line utility that helps in accessing/posting data with various protocols over the internet.** It basically serves as a bare-bones browser URL search bar. You can't render those pages like the actual GUI, and all but you can get is the HTML source code, JSON response, etc. That's still quite powerful and used in tons of applications. ``` curl URL arguments ``` The above is a basic structure of the curl command. We see the argument structure in-depth in the next section. Firstly, let's take a simple curl command with just the URL is given. ```bash curl "https://github.com" ``` From this query to `github.com`, you are literally going to `GitHub.com` and getting a response as the entire HTML source code of the page. If you don't want to spam the output in the terminal, you can redirect the output to a file. ```bash curl "https://github.com" >temp.html ``` With this command, we store the output of the command in the file temp.html, it can be any other file you like. ### Arguments It turns out that you can even parse in certain arguments to the `curl` command to get some desired and modified results. Let's take a look at some of them. The [entire list of arguments](https://curl.se/docs/manpage.html) is quite huge and baffling, but this shows how customizable the command is. - `-s` (silent the progress bar) - `-X` (web requests `POST, GET, etc` to the URL) - `-o` (output to a file) - `-H` ( provide Header to the request) - `-d` (providing the data e.g. in POST request) ```bash curl -s -o "https://github.com" temp.html ``` This command doesn't load the progress bar and simply outputs the response in a file, making the execution process in the terminal clean. ### Integration with other commands As said, the `curl` command can be well integrated with the other commands using piping in shell, assigning to variables, and so on. Let's see how we can convert the `JSON` response to a BASH variable. ```bash resp=$(curl -H "api-key: N2vDzMyEeYGTxjUTePhC8bYd" https://dev.to/api/users/me) echo $resp ``` Here, we are fetching the `JSON` response from the `dev.to` [API](https://developers.forem.com/api/),The wired string `N2vDzMyEeYGTxjUTePhC8bYd` is my [dev.to API token](https://dev.to/settings/account)(don't worry I have revoked it:) ) we have provided an argument `-H` that is a Header for accepting a `Json` response. We can store the contents of the curl command by using the `$( )` and assigning that to the variable name of your choice. ```bash username=$(curl -H "api-key: N2vDzMyEeYGTxjUTePhC8bYd" https://dev.to/api/users/me | grep -o -P '(?<=username":").*(?=","name)') ``` Here, we have stored the username from a `JSON` response to the variable username. We have piped the curl command so that we can work with that `JSON` response and modify the contents and then store the final results in a variable. In this case, we are using `grep` to filter out the content between the key `username` and `name`, thus we get the value we desired. To see the value you can always run the echo command as below: ```bash echo $username ``` So, that's how the `curl` command integrates flawlessly with BASH and other shell programming languages. ## Where is it used? `curl` is actually used in API testing, CLI applications, Web Scrapping, etc. It's a great tool for terminal lovers. Let's see where we can use the curl command actually to make some good projects. ### API Testing We can use, `curl` to test an API, it might be API you would have made or to simply test and play with other API available publicly. You can get an in-depth guide about [Testing a REST API with curl](https://www.codepedia.org/ama/how-to-test-a-rest-api-from-command-line-with-curl/). Actually, curl can do more than just testing, I have made a [bash script](https://gist.github.com/Mr-Destructive/80860664b1014ef0b94092d68ead1044) that actually posts some data over a database through the API so that I don't have to do that manually. That is the kind of automation I love to do and curl! Just did that. ### Web Scrapping Web-scrapping is usually trending with Python, but I have done that with BASH. That might be an outdated idea but is a good task to learn the basics of Web-scrapping with BASH ;). I must say that sed, awk, grep are the tools are powerful like heck in doing these tricks. I have made this [crypto-currency](https://mr-destructive.github.io/techstructive-blog/bash/2021/07/15/BASH-Crypto-Coingecko.html) and [dictionary](https://mr-destructive.github.io/techstructive-blog/bash/2021/07/27/BASH-script-dictionary-scrap.html) scrapper with BASH. Web-scrapping can be done with the curl command by fetching to an API if any or any website. We need to search and find the particular fields, classes, or ids the elements the required data might be into and then extract and filter using the tools like grep, sed or awk. ### CLI Applications We can make CLI applications like creating a terminal view of existing applications using their APIs or website. I recently made a CLI for [cross-posting articles](https://github.com/Mr-Destructive/crossposter) to dev. to, hashnode and medium. That is a project still in progress(tons of bugs) but still serving a decent job. Definitely `curl` might not be the only command that works here, but the project might look so incomplete without `curl`. **There might be other applications as well, who knows there is a lot to do with this command.** If you know one, please let everyone know in the comments. ### References: Special Thanks to the creator of the curl command - [Magnus Daniel Stenberg](https://github.com/bagder) and the developers who are still contributing and maintaining the great project. ### Conclusion So, from this article, we were able to understand the basics of the `curl` command and understand its applications in actual programming stuff. Hope you liked it. Thanks for reading and until then Happy Coding :) https://www.meetgor.com/posts/crossposter-shellscript Introduction If you have been writing articles you know the pain to get some attention, if you have already been cross-posting your articles it usually takes so Sun, 31 Oct 2021 00:00:00 UTC ## Introduction If you have been writing articles you know the pain to get some attention, if you have already been cross-posting your articles it usually takes some time to do that. This task can be automated with a shellscript. If you have been cross-posting articles on `medium.com`, `dev.to` and at `hashnode.com`, then I have a treat for you. Introducing **crossposter.sh**!! ## What is Crossposter.sh? ### Crosspost to dev.to/hahsnode/medium from the command line. Crossposter.sh is a shellscript(BASH) to automate crossposting to platforms like dev.to, medium.com and hashnode.com. The script takes in markdown version of your post with a few inputs from you and posts it to those platforms. You would require a token/key for each of those platforms to post it from the command line. You can check out the official repository of [Crossposter](https://github.com/Mr-Destructive/crossposter). The actual script is still not perfect (has a few bugs). Though it posts on `dev.to` and `medium.com` easily, the `hashnode.com` is buggy as it parses the raw markdown into the post and doesn't render as desired. So, **its a under-development script**, fell free to raise any issues or PRs on the official GitHub repo. Run the script on a bash interpreter with the command: `bash crosspost.sh` For posting the article you need to provide the following details: ## Front-Matter ### Meta data about the post - Title of Post - Subtitle of Post - Publish status of post(`true` or `false`) - Tags for the post (comma separated values) - Canonical Url (original url of the post) - Cover Image (URL of the post's image/thumbnail) This information is a must for `dev.to` especially the `title`. This should be provide in the same order as given below: ```yaml --- title: The title of the post subtitle: The description of your article published: true tags: programming, anythingelse canonical url: url of your original blog cover_image: coverimage_url --- ``` There is no need to enclose any of them with quotation marks. `Published` argument will be `true` if you want to publish it and `false` if you want to keep it in your Drafts. In the demonstrations, we just need to enter the tokens once. The tokens will be stored locally in the `keys.txt` file and retrieved later within the script. ## Posting on **dev.to**: Posting on dev.to requires their `API key` which can be generated by going on the [Dev Community API Keys](https://dev.to/settings/account/). From there you can generate a new key with any name you like. You just need to enter the key to CLI once or manually enter in the `keys.txt` file with the format `dev.to:key` on the first line. This will be used for the future cross-posting whenever you execute the shell script(`bash crosspost.sh`) You can provide the [front matter](#front-matter) manually in your markdown file or you will be prompted for the input. So, that is all you will require for posting on dev.to from the Command line. Lets see the script in action  If you want to add in more stuff to the post, you can check out the [DEV.to API docs](https://developers.forem.com/api#operation/createArticle) which is powered by [Forem](https://www.forem.com/), there a ton of options you can hook to the front-matter in the shellscript. **NOTE: There is a limit of 10 requests per 30 seconds, so keep in mind while testing the script and don't try to spam** ## Posting on **hashnode.com**: This part is still under development as it only displays the raw markdown in the post, also the `tags` are too heavy to implement from the API as `id` of every tag is required along with the `slug` and `name`. Still it serves some purpose at least. For posting on `hashnode.com`, we need `Personal Access Token`. This can be generated by going to [Developer Settings](https://hashnode.com/settings/developer). You will also require the user-id of your `hashnode` account. You can get your user-id/username from the [settings](https://hashnode.com/settings) tab under profile information. We require Username for posting to the Publication Blog if any. As usual, the `Personal Access Token` for interacting with the [Hashnodes' GraphQL API](https://api.hashnode.com/). The API is quite user friendly and provides everything in one place. There are docs for running each and every `query` and `mutations` present in the API. You can paste the token when prompted from the script or manually type in the `keys.txt` text file as `hashnode:token` on the 4th line. Yes, that should be on the `4th` line, thats make retrieving much more easier and safe. Next also input in the `username` when the script asks for the input or again type in on the `5th` line, `hashnode_id:username` in the text file `keys.txt`. Please enter the credentials from the script prompt so as to avoid errors and misconfigurations when doing manually This will create the Post on hashnode with the title, subtitle, cover image correctly but will mess up the content. I tried hard but its just not happening. There needs to be some character for newline as the API rejects the `rn` characters passed in, so I have substited them with `br` and the result is raw markdown. **As the Hashnode API is still under development and they are bringing changes and new features in, the API should improve in its core functionality and make it much easier for creating some common queries**. So, I'll create issue on GitHub for posting the actual content via the script. So, this is the demonstration of posting on hashnode.  ## Posting on **medium.com**: Medium API is much more versatile and markdown friendly, though it has some limitation on the number of posts you can make in a day. For posting on `Medium.com`, we will require the `Integration Token` which can be generated on the [settings tab](https://medium.com/me/settings). As similar to `hashnode`, you can name the token whatever you like and then get the token. Paste the token when prompted from the script or manually type in the `keys.txt` text file as `medium:token` on the `2nd` line. We also require the Medium_id, but we can get that from the token itself, so inside the script once the token is obtained, the curl command is executed to fetch in the `id` and it is stored on the next(`3rd`) line in the `keys.txt` file for actually posting on `medium.com`. So that is all the configuration you need for posting on `medium.com`. There is some documentation on [Medium API](https://github.com/Medium/medium-api-docs), we can even post to a Publication, that shall be created in future. Also the cover images can be posted on medium, it is not currently done but that can again be a #TODO. **The tags are not rendered on Medium yet with the script.** The way we can parse strings is limited in BASH, so this might still be a doable thing later. Most of the checkboxes are ticked like title, subtitle, cover-image, canonical url, and importantly the content. Let's look at post on medium from the script.  ## All platforms: Now, once you have configured every thing, you can opt for the `4` choice that is post on all platforms(dev.to, hashnode and medium), but as hashnode is not looking a good option right now, so there is the `5` option for only `dev.to` and `medium`.  ## Why use Crossposter.sh? This can be not so big of an issue for most of the people but it was a good side project to work and learn more about how APIs work and get some ideas on the design of the platform. Though it is quite time saving to cross post on 3 different platforms within a minute or two. You can tailor your own script as per your specifications and desire. So, if you are an author on all of the mentioned platforms, please give it a try. Other Platforms are welcome for contributions. If you found any unexpected things, please hit them in the `issues` tab. ## Script The script mostly leverages `curl`, `sed`, `cat` and some other basic utilities in BASH. ### Using `curl` for posting the article from APIs Curl is a life saver command for this project, without this tool, the project might not be as great and efficient. Let's see some quick commands used in the script. ```bash curl -H "Content-Type: application/json" -H "api-key": \"'"$key"'\" -d '{"content":\"'"$body"'\"}' "$url" ``` So, the above command is quite basic, some more additions are also added as per the specifications of the Platform. But, let us understand the structure of the command we are sending to the APIs. The first part is the Header (`-H`), in here we specify the content that is going to get parsed and the api-keys to access the API. Next, we have the body or the data (`-d`), here we parse in the actual contents, it might be the front matter along with the markdown content. Finally we have the `url` where we send the `POST` request i.e. the `API endpoint`. The is the escape character that is used to preserve the literal value of the next character and in short we can shorten the command to fit in the next line. The wired `$body` is used to parse the value of the variable `body` inside of single quotes as in BASH, we can only access the variables' value in double quotes. We are using single quotes as we have to pass the `json` object and which has already double quotes in it. ### Using `sed` for editing text Sed is a super-powerful stream editor, its somewhat similar to Vim without an interface, only commands. We use this tool to manipulate the front-matter for posting on the platforms by parsing them to variables in BASH. We also use to enter the api keys inputted by user from variables into the file at a specific position to retrieve later. ```bash sed -i "1a title: $title" file.txt ``` Here, we are appending(`a`) to the 1st line, text `title: $title`, here `$title` is the variable, so we are technically parsing the value of the variable `title`. We are editing the file `file.txt` in-place `-i` i.e. we are editing it live without creating any temp or backup files. ```bash sed -n -e "s/dev.to://p' keys.txt" ``` Here we are essentially getting the text after a particular pattern. In this case we are searching in `keys.txt` file for the string `dev.to:` and anything after that till the end of line is returned, we can further store it in the variable and do all sorts of operation. ### Using `awk` for programmatic editing ```bash awk '{print $0" "}' temp.txt >file.txt ``` AWK is a command-line utility for manipulating or writing certain operations/patterns programmatically. We use this tool so as to add `4r4n` to the end of each line, the APIs can't parse the file contents directly so we have to add certain characters before the end of line and do further operations. ```bash cat temp.md | tr -d ' ' > temp.txt ``` After we have added the ` ` characters to the end of the file, we simply can use `cat` and `tr` to merge all the lines into a single line. This is how we parse the contents to the API more safely and concisely, of course we need to parse them in a variable by reading the file. OK, I won't bore anyone with more BASH but that were some of the quite important commands in the script that form the backbone of the cross-posting and handling text with the APIs. ## Conclusion So, we can see `crosspost.sh` is a BASH script that cross-posts markdown articles with a bit of inputs to 3 different platforms within a couple of minutes. This article was basically to demonstrate the project and its capabilities also highlighting the issues present. I hope you liked the project, please do try it and comment the feedback please. Thank you for reading, Until next time, Happy Coding :) https://www.meetgor.com/posts/hacktoberfest-2021 Introduction This was my second year as a Hacktoberfest participant, First year was very dry and not so exciting as this turned out to be a cracker of an experi Thu, 28 Oct 2021 00:00:00 UTC ## Introduction This was my second year as a Hacktoberfest participant, First year was very dry and not so exciting as this turned out to be a cracker of an experience. With more than month gone in with only a few contributions, the second half of October was a splendid one for me where I made a lot more than in entire 2 years. This is my Contributions so far in this month:  ## Maintaining the Momentum Since the beginning of September, I started to contribute to open-source little by little so as to really progress my way into learning new stuff and networking with more people. I also had Hacktoberfest in my mind and so I started with simple issues and got a different perspective of Open-source as it is another way to do things pretty quickly and feeling of getting things done for someone else as a community work. Finding Issues perfect for you is hard but shaping yourself with the requirements for the issue is what you can achieve over a period of time. So I basically had some 5-6 pull requests merged in some Repositories, I don't trust numbers when it comes to open-source as it might be some `documentation changes`, `branch name changes`, `translations`, etc. This might not be the best contribution I could have made but breaking the ice was an important step, for that I bought my expectations down to contribute to big organizations and projects and started sifting some doable issues. Since then, Hacktoberfest was around the corner and it just carried the momentum of my open-source contribution further. ## Getting the hang of Open Source So, till mid-October, I just had 2 pull requests with only 1 merged. I was out of town for a week and had some college projects to be completed so I couldn't catch up with the contributions but after a few days of getting back on a regular schedule, I was able to submit half a dozen of PRs in a week (all merged). One of them was from the official `Jekyll` repository :) I was pretty much elated with that contribution though it was only about adding 4-5 lines of documentation, it was something that can be useful for people as it was for me and it was not previously there in the official docs. You will say, is open-source about merging PRs? No, it's not especially the `hacktoberfest` labeled ones. I have contributed to only 5 projects with hacktoberfest accepted label others were the stuff which I personally liked and wanted to genuinely contribute to. I also would like to address some issues and things which are not in the spirit of Open source, firstly **most of the issues with hacktoberfest accepted labels are from front end, so trivial, self-assigned. I understand it's meant for beginners but it's not only for beginners or is it? I might be wrong here.** Still, the maintainers need to up the standards a bit for equal opportunity for everyone to contribute with the spirit of open source. I wanted to be a maintainer this year but didn't feel much confident about it. It can be a challenge for maintainers as well to fetch attention and come up with good issues and projects. That's a thought for another year until then I will be a happy contributor. > ### Hacktoberfest is only about Open Source but Open source is not only about Hacktoberfest Well, Hacktoberfest is about open-source right? But Open source is not only about Hacktoberfest, like just contributing to getting 4 PRs merged and just forgetting open source because it was mentioned so(also get some Swags). This was my mistake last year :( I just didn't touch open-source after Hacktoberfest. This year I would definitely not repeat the mistake and continue providing value and gain more experience. ## What were my learnings I learned some skills that I might not have otherwise thought about learning. Understanding `Svelte`, `Github Actions` for `Go` lang, leveraging `Django` and `Bash` skills were some of my key takeaways from this year of Hacktoberfest. This is too much for me compared to the previous year in which I just used some `C++` and `Bash` to create an Algorithm or function. How stupid of me for contributing to that kind of project and pulling up 4 PRs for that, though I was a beginner and have come too far from making that mistake again. Thanks to everyone who gave me a chance to contribute. Let's see my top 4 contributions as per my opinion: - [Created a ripple effect button as a `Svelte` component](https://github.com/Ananto30/golpo-svelte/pull/14) - [Created a BASH script to convert a pip list to freeze format output](https://github.com/soumya997/Listreqs/pull/2) - [Added `GitHub` Actions workflow for linting in `go`](https://github.com/julien-bouquet/geo-api/pull/18) - [Additional Documentation for running Jekyll local](https://github.com/jekyll/jekyll/pull/8852) ## What's next in Open Source for me I'll continue to sort and pick up some issues in my spare time and try to learn something that I have never worked with or am too excited to work on. I have a lot of plans for some Open source projects of existing applications and would like to make them from the ground up. Some of the things I'm looking for in open source next: - Open-source alternative to existing projects/apps - Any new application/project - Automation of tasks (using Python/BASH) - Writing documentation for existing projects Don't take my word for it, you'll see it in action soon(probably in 2022) XD. Until then it's about learning and documenting the process. ## Conclusion So, it's almost Halloween with Diwali around the corner and sadly an end-of-a-month-long celebration of Open source. Though you can anytime contribute to open source this is a month you can do more than just open source. This year for me was an absolute delight and definitely improved from the horrendous previous experiences. The things I spent time learning in lock-down when there was no result/reward for the effort, are finally getting rewarded, and am much confident than I ever was. Hopefully, you have not read the entire article but that's fine unless you have taken time to read it. Thank you for being here. Until then Happy Coding :) https://www.meetgor.com/posts/flask-django-frameworks Introduction We all have seen the buzz around web frameworks like Django, Flask, Node.js, etc but have you taken time to learn all of them? No, and you shouldn' Sun, 24 Oct 2021 00:00:00 UTC ## Introduction We all have seen the buzz around web frameworks like Django, Flask, Node.js, etc but have you taken time to learn all of them? No, and you shouldn't! Because many web frameworks share the same principle and workflow with a bit of difference. It's just like learning one programming language and applying the same concepts in a different syntax and mechanism. In the world of web frameworks, this is the case as well, but most of them will disagree with it as every web framework is unique in its design and that's true, don't get me wrong. Before we get into frameworks let us understand the key components of the web application - **Database** - It holds the data for our application. - **Server** - Used to fetch/store/manage requests from the client. - **API** - Used as an interface between the client and the Database. - **Client** - The browser or any client that requests for resources.  Every web framework will serve the same purpose with different design, architecture, language but it will have a similar pattern in developing the application. Let's clear the concepts in this article. ## What is a back-end Web framework? A web framework is a tool/application meant for designing, creating, testing web applications a lot quicker with a lot of ease. Without web frameworks, you will have been writing some code that will tire you very quickly. It even seems impossible to manually write markups for each piece of data in the application, which is taken care of by dynamic templating in Python-based frameworks like Django, Flask. The database queries are managed by the web frameworks as well, otherwise, you will have been writing SQL queries manually! How painful and frustrating that would look, of course, you can create scripts for querying to the database but you are then creating a component of a framework. `Don't waste time` that's a takeaway from the philosophy of all the web frameworks. Another thing that back-end web frameworks do is create homogeneity in development across different environments and applications. It also creates a developer-friendly environment. We must not forget how easy and quick applications can be built using the back-end web frameworks. ### A back-end Web framework provides some of the features like: - Handle web requests - Manage DB by just using some simple scripts - Render Dynamic Templates - Provide a lot of native-language libraries integration - Organize a project much easily and effectively - Options to scale the application at any level - Provide some standard and secure way to run the server(production) - Design APIs much easily Let us look at two of the most popular frameworks in the Python community. ### 1. Flask ### 2. Django  We are seeing that Django and Flask are among the top 3 back-end web frameworks in 2021. So there is no double thought on why you should learn these technologies. ### What is Flask Flask is the bare-bones framework that provides a lot of customizability with a lot less boilerplate code. It is a framework that provides a lot of third-party libraries to add functionalities to our application. > Flask is a micro web framework Flask as per the official documentation is a `micro` framework indicating it has a very minimal setup. It is a back-end web framework that can be structured as per needs with a very little configuration overhead. That being said, it can get a bit limited in structuring and functionalities as it needs to taken care of manually. **Flask is the easiest back-end web framework to get started and learn the fundamentals of server-side**. Flask is quite flexible in terms of scalability and maintenance of decent-sized applications as well. Though the community is not that big and absence of standardization in Flask, it is a go-to back-end web framework for beginners as well as experts due to its simplicity and flawless integration with Python libraries. The main concepts in Flask might be: - Virtual Environment - WSGI as a web server - App routing - Jinga2 as a templating language - Creating Database connections So, **Flask is kind of a DIY back-end web framework with rich sets of libraries and customizability out of the box**. This can easily be a beginner's choice and a right one too. ### What is Django Django is also a back-end web framework based on Python programming language but it is more standardized and high-level. Django encourages a defined pattern for development but with customization and freedom in mind. Django also modularizes the components into so-called `apps` to provide a scalable experience. It has a lot of boilerplate code to get up and running quite easily, it also has a `Admin section` pre-built with all the functionalities. Similar to `Flask`, it also provides flawless integration with all the Python libraries. It provides a much easier Database integration and pre-built `User` authentication along with its model ready to plug in and use. > Django is a Batteries included Framework That means it has baked in functionalities like User-Authentication, Admin Section, Database Integration, RSS/Atom syndication feeds, etc.  The main concepts in Django include: - Virtual Environment - WSGI/ASGI as web servers - Project structure - `Model View Controller` Architecture in developing apps - Django Templating Language for rendering Dynamic Templates - `Object-Relational Mapping` in creating the applications Unlike Flask, Django is already baked in with a lot of functionalities and integration with a ton of features. It should be good for beginners but many things are already taken care of that can be a huddle in **actual learning process**, that being said it is a much scalable and production-ready web framework (not only back-end). ### What are the similarities between them? Well, if you learn one the other will be quite easy enough to pick up. The overall development is almost similar but unique in its own way. - Pythonic syntax and libraries - Project Structure is quite similar to `blueprints` in Flask and `apps` in Django - Templating Language is almost similar - Static Files are handled similarly with a different syntax - URL Routing is the same as it binds the view(functions) with a pattern - Ease in Deployment with minimal configuration ## What should you learn? That question is dependent on the type of application you are trying to make but for a beginner trying to get hands dirty on the server-side, I would recommend `Flask` as it is quite minimal and helps in constructing the base for the concepts like APIs, Databases, Requests, Admin section, etc. This might not be that difficult for people trying to learn back-end from scratch but for people with a bit of programming and server-side experience, `Django` should be a go-to framework for all their needs. At the end of the day, it hardly matters what you do with which framework, what people see is the end result. ## Conclusion Thus, from this article, you might have got a bit understanding of why are the frameworks used in making applications and also the similarities and differences in the Python-based back-end web frameworks like Django and Flask. If you have any thoughts please let me know in the comments or on my social handles, any kind of feedback is much appreciated. Thank you for reading till here, until then as always Happy Coding :) https://www.meetgor.com/posts/vim-un-comment-p1 Introduction We as programmers always fiddle with commenting out code for code testing, documenting the function of code, and most importantly debugging. So you Thu, 07 Oct 2021 00:00:00 UTC ## Introduction We as programmers always fiddle with commenting out code for code testing, documenting the function of code, and most importantly debugging. So you can't wait to comment on a large chunk of code manually, as it is quite a tedious thing to do. Let's do it effectively in Vim. In this part of the series, I'll cover how to comment/uncomment chunks/blocks of code effectively in Vim. We will see and use some commands, keybindings for doing so, and also we would add certain components to our vimrc file as well to design some custom key mappings. Let's get faster with Vim. ## How to comment multiple lines effectively To comment on multiple lines of code, we can use the Visual Block mode to select the lines, and then after entering into insert mode, we can comment a single line and it would be reflected on all the selected lines. 1. Press `CTRL+V` and Select the line using j and k 2. After Selecting the lines, Press `Escape` 3. Press `Shift + I`, to enter insert mode 4. Enter the comment code (`//`, `#`, or other)  So, using just simple steps you can comment out large chunks of code quite easily and effectively. If you are using some other language that has multiple characters for commenting like `//`, `- -`, etc, you can type in any number of characters while being in insert mode after selecting the lines.  This might look a bit wired on the first try but just try it every day, It is a life-saving and very satisfying experience once applied in a real-world scenario. ## How to uncomment multiple lines effectively Now, as we have seen to comment out a large chunk of code, we can even uncomment the code very easily. It's even simpler than commenting the code. 1. Press `CTRL + V` to enter Visual Block mode 2. Select the commented characters 3. Press `d` to delete the comments 4. Press `Escape`  We can simply use the CTRL + V to select the comment, and then press d to delete all the comment characters. **We are using the Visual Block mode as we only want the comment to be selected and not the entire code associated with the lines.** ## Using Multiline Comments for Programming languages Now you might say, why use multiple single-line comments when we can use multiline comments in almost all programming languages. Well, Of course, you can do that, it's easier for reading the code if syntax highlighting is accurate and greys out the commented part. We can simply add those characters to the start of the block and at the end of the block. But in Vim, we can customize that too, just imagine when you just select the chunk/block of code that you need to comment out and then simply press a few keystrokes (just 2) and the multiline comments are automatically (programmatically) added as per the programming language extension of the file. Isn't that cool? Well, you just need to copy-paste the below code to your Vimrc file and source it and you are good to go. ```vim function! Comment() let ext = tolower(expand('%:e')) if ext == 'py' let cmt1 = "'''" let cmt2 = "'''" elseif ext == 'cpp' || ext =='java' || ext == 'css' || ext == 'js' || ext == 'c' || ext =='cs' || ext == 'rs' || ext == 'go' let cmt1 = '/*' let cmt2 = '*/' elseif ext == 'sh' let cmt1 = ":'" let cmt2 = "'" elseif ext == 'html' let cmt1 = "<!--" let cmt2 = "-->" elseif ext == 'hs' let cmt1 = "{-" let cmt2 = "-}" elseif ext == "rb" let cmt1 = "=begin" let cmt2 = "=end" endif exe line("'<")."normal O". cmt1 | exe line("'>")."normal o". cmt2 endfunction function! UnComment() exe line("'<")."normal dd" | exe line("'>")."normal dd" endfunction vnoremap ,m :<c-w><c-w><c-w><c-w><c-w>call Comment()<CR> vnoremap m, :<c-w><c-w><c-w><c-w><c-w>call UnComment()<CR> ``` The below screencast is an example of `HTML` snippet in a file that is getting commented using mapping with the keys `,m` you can put any other keybinding you like.  --- Similarly for the next screencast is of an `Javascript` snippet in a file which is getting commented using a mapping with the keys `,m` and uncommented using `m,` again you can put any other keybinding you like.  --- The following screencast is of a shell script(BASH) snippet.  --- ### Multiline Comments in various Programming Languages: #### 1. C / C++ / Java / Javascript / CSS / C# / Rust / Go / PHP / Swift / Dart / Kotlin ``` /* */ ``` #### 2. Python ``` ''' ''' ``` You can even use `"""` double quotes instead of single quotes #### 3. BASH (Shell Scripting) ``` : ' ' ``` You can even use `: "` and `"` double quotes instead of single quotes #### 4. Haskell ``` {- -} ``` #### 5. Ruby ``` =begin =end ``` #### 6. HTML ``` <!-- --> ``` #### 7. Julia ``` #= =# ``` ### Understanding the Commands / Keymapping **NOTE : You need to go from the top to bottom while commenting on the block of code, otherwise, it would be a mismatch in commenting for specific language syntax. While uncommenting the order doesn't matter.** #### Getting the extension (filetype) In Vim, we can get the file extension i.e. we can get the programming language associated with the current file. To do that we can use, `expand('%:e')`. This will give us the file extension of the current file. Just for simplicity, `%` means the current file, added to it is `:e` for excluding the filename and keeping the extension. We convert the extension into lowercase just for keeping things safe and programmatic and store it in a variable `ext`. #### Checking for programming language We then can then use an if-else ladder to check for the programming languages and assign two variables `cmt1` for the initial characters in the multiline comment and `cmt2` for enclosing the comment. #### Typing in the characters We can use the function `line("'<")` to get the line number of the previous visual selection. Similarly, `line("'>")` for the ending line. We are using the `exe` command to execute the function `line` and so we have to use a concatenation of the commands even to write the raw commands like `i` to insert mode, `o` to insert mode but a line below the cursor. So, we use `normal` command for that. This command indicates the interpreter to execute the following commands from the normal mode. We have to enclose the `normal` command in double-quotes/single quotes. We can simply use the variable again with concatenation. ```vim exe line("'>")."normal o". cmt2 ``` The above command will fetch the last line's number of the previous visual selection followed by entering `o` from the normal mode and concatenated with the value of the variable `cmt2` which we have already initialized in the `Comment` function. We are using `|` for running multiple commands as we also need to include the comment at the beginning of the visual selection. For uncommenting the code, we are simply deleting the entire first and the last line in the visual selection. For that, we have used `dd` from the normal mode. ### Conclusion So, from the following type of tutorial, we were able to set up our Vim editor for efficient code commenting/ uncommenting using some commands, key shortcuts, and configuring the vimrc for making custom keymappings. We were also able to understand the multiline comments in various programming languages and use them in Vim very effectively with a simple addon to the config vimrc file. Thank you for reading, hope you found this article helpful. If you have any queries or wanna add multiline comments for some more programming languages please let me know in the comments or contact section. Happy Coding :) ### References - [StackOverflow - Commenting lines in Vim ](https://stackoverflow.com/questions/1676632/whats-a-quick-way-to-comment-uncomment-lines-in-vim/1676690) - [Liz Lam - 3 ways to comment code in Vim](https://dev.to/grepliz/3-ways-to-comment-out-blocks-of-code-in-vi-6j4) - [StackExchange - Use variable in normal command](https://vi.stackexchange.com/questions/9644/how-to-use-a-variable-in-the-expression-of-a-normal-command) https://www.meetgor.com/posts/ml-intro Introduction We all hear about terms like , , and others, but what do they actually mean and why do you need to care about these as a developer. This won't be a Wed, 29 Sep 2021 00:00:00 UTC ## Introduction We all hear about terms like `Machine Learning`, `Artificial Intelligence`, and others, but what do they actually mean and why do you need to care about these as a developer. This won't be a perfect guide in terms of experience but surely enough to get anyone through the basics of Machine Learning. This is not the kind of article I write but, having such challenges can help me become a better technical writer, this is the challenge put forward in the Hashnode Bootcamp 4 to get out of my comfort zone. Here's my take on what I know about Machine Learning till now (P.S. Half of the stuff I discovered and re-learned during writing). ## What is Machine Learning? Machine Learning is a technique in software development to predict and react to the inputs without being explicitly programmed or written. We can use the if-else condition till a point in time, after seeing real-world examples like customer service, driving, playing games(chess, checkers, etc), image prediction, and so on. You can't write code for every single case of these applications, So that is where we see Artificial Intelligence. > Artificial Intelligence is a process of simulating human-like behavior into computers /robots/ electronic systems. These are two quite similar terms(A.I., M.L.) but they have their own differences. Let's look at those differences: - **Artificial Intelligence is a technology that enables computer systems to act and decide like humans.** - **Machine Learning is a process of extracting data and learning from the past experience or outcomes from that data.** Machine learning is actually a subset of AI. Machine learning actually is about training the computer system about an outcome using the data feed into it. We will actually look at the detailed process about it in the next few sections. ## The Process of Machine Learning The first step should be to choose an idea or a goal that you would like to make the system predict or output the results. 1. Data Gathering 2. Filtering Data 3. Selecting an Algorithm 4. Training the system 5. Verifying and Evaluation of Training 6. Improving and Deploying the Model Let's take the example of classifying a picture as either dog or cat.  ### 1. Data Gathering We can now move on to collecting the medium of data that will be used by our system to get the desired outcome, it might be to predict something, classify certain things, take decisions, etc. In our example, as we want to classify an image into either a dog or a cat is classifying things from the given data set. For that, we will require images that depict this constraint. We can use our personal data, public data and from other sources, you would like to get your hands on. These are some of the popular places to get data publicly: - [Kaggle](https://www.kaggle.com/) - [Google Data Search](https://datasetsearch.research.google.com/) - [Reddit Datasets](https://www.reddit.com/r/datasets/top/?sort=top&t=all) - [Public Datasets on GitHub](https://github.com/awesomedata/awesome-public-datasets#machinelearning) - [AWS Registry of Open Data](https://registry.opendata.aws/) ### 2. Filtering Data After you have collected the data from some sources, you will notice that it is not perfect as per your needs. And to be honest, there is no dataset that is perfect for your requirements, because otherwise there will be a ton of data to work with, it might be inefficient for humans to create and sort out from that data. So we may have to do it manually or take help from a data scientist. But if you are just learning, it will be helpful for you to filter and clean the data yourself. There will be things in the data sets missing or there will be unwanted things in it. This is a critical step that everyone tends to ignore but at the end of the day, spend about 80% of the time unknowingly. This is quite an important step as it decides the efficiency of the model you will have made. - Remove/Fill in the rows which are empty. - Remove the columns which are not related to your objective. - Fix certain wrong or inconsistent data. #### Group data as Training and Testing After the procedure has been applied, you can now separate the data set as Training and Testing Data. You have to create two datasets from one, the prior for training and the latter for testing the model system after evaluating the tests. For our example, we have to separate the images which will be relatively easier to distinguish in the training data, and the tough ones for the testing data as it will challenge the model appropriately. ### 3. Selecting an Algorithm Now, this is again an important step as it will make your project's backbone. This will be the algorithm that will identify, predict or decide on the outcomes from the data given to it. We have the following types of algorithms - Linear Regression - Logistic Regression - Decision Tree - Artificial Neural Network - k-Nearest Neighbors (KNN) - k-Means You can choose any one of the above or find other types which will be more or less based on these algorithms. This algorithm will be decided by the outcomes you want, for example, whether you have to predict, classify, recommend, cluster, etc. the outcome from the given data. Different algorithms have different complexity as they have a completely different approaches. You can research this more and find more about which will be suitable for your objectives or application. Now an important topic that is misleading, Model is the program that will work with the data in association with the algorithm and output the actual objective. Model is not the algorithm but it works with the chosen algorithm and processes the actual learning in machine learning. So, > Model = Algorithm + Data The model will actually process the data according to the algorithm given and fill in the objectives may they be classifying or predicting. ### 4. Training the system Training is a step that is very interesting as it involves actually testing the model and it's really fun. We provide the model the `training data` that we segregated while filtering the data. In this process we try to minimize the loss by making changes to the algorithm, fix some data set or bring in some additional dataset as per needs and again evaluate the results. This is a loop called `model fitting`. This step depends on the learning into consideration, whether you want to provide any supervision or not. ### 5. Verifying and Evaluation of Training This is a part of `model fitting` as it is the part of the loop which allows us to evaluate and verify the model. We can evaluate the model based on its accuracy, precision, labels, etc. So based on those parameters we should be able to decide its complexity and performance. These are important aspects to consider in evaluating the model. - Accuracy - Precision - Recall You can get the details of the mathematics and logic involved in evaluating the model with some references like: - [Evaluating ML model - Jeremy Jordan](https://www.jeremyjordan.me/evaluating-a-machine-learning-model/) - [Ways to Evaluate ML model - Towards DataScience ](https://towardsdatascience.com/various-ways-to-evaluate-a-machine-learning-models-performance-230449055f15) ### 6. Improving and Deploying the Model This might be the final step generally but it depends on the project, there are certain aspects that need to be taken care of like: - Creating an API endpoint - Analysis and Visualization integration with client-side (web/android/ios/desktop app) - Creating a Pipeline for data input and output from the model. There might be other options like CI/CD, Testing, feedback, and other production level details that need to be taken care of, you can read more about the deployment of machine learning models [here](https://christophergs.com/machine%20learning/2019/03/17/how-to-deploy-machine-learning-models/). You can learn about deploying an ML model for your learning and testing for free with the recommendations of [FCC](https://www.freecodecamp.org/news/deploy-your-machine-learning-models-for-free/). ## Different Types of Machine Learning There are four basic types of Machine Learning : ### 1. Supervised Learning In this type of ML, the model is given the labeled data in the training dataset and is evaluated. We provide both input and output to the model and hence it is supervised or tracked throughout the process.  The above image is just for reference and not directly a model, height and weight can be a parameter to consider and are not only the thing to be considered her. It's just for making understand the concept of the learning process. ### 2. Unsupervised Learning In this type of machine learning, the model is trained with unlabeled data. It is on the algorithm to actually see the pattern or logic in the dataset provided and give the output. The output will be known to the user but is not given to the model, hence called unsupervised learning.  ### 3. Semi-Supervised Learning As the name suggests, it is a combination of both Supervised and Unsupervised learning. The dataset is given with the label but the model is also allowed to process its own label(kind of) into the output. Hence having the best of both worlds. There might be even some labeled and some unlabeled datasets as per the requirements of the application.  ### 4. Reinforcement Learning In reinforcement learning, the model learns from feedback. It might look similar to supervised learning but here the feedback might not be instant and hence causing delay and improper decision making from the model. Though it is used in many places, it is like the realistic learning behavior of humans depicted to computers with this learning system. **There are other types of learning systems as well but these are used quite commonly and are quite versatile as well.** ## Applications of Machine Learning The applications of Machines Learning are all around you. Just look carefully, you would have even feedback on a model! - Personal Assistants (Google Assistant/ Siri/ Alexa) - Gmail Inbox filter. - [Youtube Video Recommendation system](https://towardsdatascience.com/how-youtube-recommends-videos-b6e003a5ab2f) . - Face recognition ( [DeepFace](https://en.wikipedia.org/wiki/DeepFace) ) - Product Recommendations. - Self-Driving Cars( [Tesla](https://www.tesla.com/autopilot) ) - Traffic Alerts (Google Map) - Text Improvement ([Grammarly](https://www.grammarly.com/blog/how-grammarly-uses-ai/)) This list is quite huge and is increasing every day with new technologies and growing popularity. ## Can GitHub Copilot take away developers' jobs? This just doesn't focus on developers, it's every human's job on target this day but really? Is it a matter of concern? I don't think so, because, > the number of jobs lost = the number of jobs given. There will be a need for humans in some or the other way, remember a computer cannot is **not smart** like humans, surely it has improved from what we thought a couple of years ago. But who discovered this? HUMANS. Yes, Machine Learning is quite a powerful technique but humans will remain the essence in the world. It will be dependent on humans how we treat the models and use them to our and nature's advantage and not use them against nature to face the consequences later. ## Conclusion Ok, so from this big article, we can summarize the Machine Learning concept. > Machine learning is a program that has a dataset and algorithm along with the model for the objective, we train the model as per the requirements and objectives with our dataset. We were able to understand some common processes involved in Machine Learning. We even discussed the applications and the state of Machine Learning in today's world. I hope you found this article helpful. Thank you for reading. Happy Coding :) https://www.meetgor.com/posts/djagno-auth0-script Introduction This is a guide and a walkthrough of how to quickly set up a base Django project with Auth0 as integration for authentication and authorization. I Tue, 28 Sep 2021 00:00:00 UTC ## Introduction This is a guide and a walkthrough of how to quickly set up a base Django project with Auth0 as integration for authentication and authorization. I will walk you through the Django setup and how to use and integrate the functionalities of the Auth0. I will also discuss how why you should be using Auth0 and why I love it. The script takes `2:44` minutes time to do everything from scratch. From installing virtualenv in python to integrating the Auth0 application. Here's how the script works:  ### Contents - [What is Auth0](#what-is-auth0) - [Why I love Auth0](#why-i-love-auth0) - [Set up a Django Project](#set-up-a-django-project) - [Integrate Auth0 to a Django project](#integrate-auth0-to-a-django-project) - [Creating a BASH Script for integrating Auth0](#creating-a-bash-script-for-integrating-auth0) - [Appending to a file](#appending-to-a-file) - [Adding text before a particular line using `sed` ](#adding-text-before-a-particular-line-using-sed) - [Appending to a line using `sed`](#appending-to-a-line-using-sed) - [Complete BASH Script](#complete-bash-script) - [Conclusion](#conclusion) ## What is Auth0 Auth0 (`Auth zero`) is a platform that provides easy authentication and authorization for a number of platforms in various programming languages and frameworks. The easy-to-follow documentation, availability for almost all web frameworks across platforms make it a big bonus for developers. They actually make the Developer experience flawless and beginner-friendly. According to Auth0, > They make your login box awesome And how true is that they make things pretty convenient and wicked fast to integrate a smooth functional backend for authentication and authorization. Of course, there are more things they offer than just making authentication systems but it is by far what the world knows them for. ## Why I love Auth0 Auth0 is a generous company that provides a free tier for a limited capacity of authentication and that might be more than enough for a developer getting its feet wet in the web development (backend). They even provide a nice user interface out of the box for login/signup and even a dashboard ready-made, which is quite a lot of heavy lifting already done for you. Also, there is a dashboard for analyzing the number of sign-in/logins into the particular app. This provides the admin/developer of the app to get a closer look at the user registered in a day/week/months, number of active users, and so on. So, who would not love it? I am willing to write and use their service for some of my projects. I already have used one for the Hashnode x Auth0 Hackathon, I made [devquotes](https://github.com/Mr-Destructive/devquotes) using the authentication of Auth0 in my Django application. ## Set up a Django Project If you are reading this you already know how to set up a Django project, I assume. But nevertheless, I can just include a quick introduction on how to do it. I have a script to do this. ```bash #!/usr/bin/env bash mkdir $1 cd $1 pip install virtualenv virtualenv env source env�in�ctivate pip install django django-admin startproject $1 . clear ``` You can check out [Django Quick Setup Script](https://techstructiveblog.hashnode.dev/django-quick-setup-script) for the details of this script and also a more in-depth guide of Django project setup. But if you want to understand the basics of the Django project setup here is a little guide about it: Firstly, create a virtual environment, it's not mandatory but it keeps things simple and easy for your project in correspondence to the entire OS. So in python, we have a module to create the virtual environment pretty easily, ```shell pip install virtualenv ``` You can use `pip3` or `pip -m`, or however you install normal python modules. This just installs the python virtual environment, we need to create one in the current folder, so for that navigate to the folder where you want to create the project and enter the following command: ```shell virtualenv venv ``` Here, `venv` can be anything like `env` just for your understanding and simplicity it's a standard name kept for the same. After this, you will see a folder of the same name i.e. `venv` or any other name you have used. This is the folder where python will keep every installation private to the local folder itself. Now, we need to activate the virtual environment, for that we can use the command : ```bash # for Linux/macOS : source venv/bin/activate ``` ```batch # for Windows: venv\Scripts�ctivate ``` After this, your command prompt will have a `(venv)` attached to its start. This indicates you are in a virtual environment, things you do here, may it be module installation or any configuration related to python will stay in the local folder itself. After the virtual environment is set up and activated, you can install Django and get started with it. Firstly, install Django using pip: ```shell pip install django ``` After the installation is completed, you can start a Django project in the current folder using the command: ```shell django-admin startproject name ``` Here name can be your project name. After this, you will see one new folder and one file pop up. Namely, the `project named` folder and `manage.py` file. So you don't have to touch the `manage.py` file but we use it in most of the commands to use the Django functionalities. You can now run your basic server using the command : ```shell python manage.py runserver ``` There is a base installed/setup of the Django project. Moving on in integrating the Auth0 login functionality in our webpage. ## Integrate the Auth0 app in your project So, for integrating the Auth0 app for your web application, you need to have an Auth0 account, you can signup here. After this you can create an Auth0 application for any type of application, we have a couple of options: - Native Application - Single Page Application - Regular Web Application - Machine to Machine Application  You can choose as per your needs, but mostly it would be a Regular Web application if you have a backend like Django, Nodejs, PHP, or other types of frameworks and languages. So, moving ahead we have created an application for the Django framework as a backend. Now, we have a `Settings` tab in the application dashboard, where we have different credentials for the Auth0 app to talk to our application. The credentials needed to be stored safely are: - domain - Client ID (Key) - Client Secret This has to be secured for our local application which will go into production when ready. You can use several options like dotenv, environment variables, and so on when the application is being deployed but for now, let's hardcode them in our Django project. Now, you can follow the simple straightforward procedure to copy-paste your credentials from the [Auth0 official documentation](https://auth0.com/docs/quickstart/webapp/django/01-login#logout). It's quite straightforward to follow the steps even for a beginner. After the Auth0 app has been configured following the procedure in the documentation, you need to integrate several files like dashboard and index templates into your custom templates. Following additional changes are also to be made if you have a user-defined app for your Django project. In the `auth0login` app, `view.py` file: 1. The `index` function renders the base file for your project if the user is logged in. 2. The `dashboard` function renders the baked version of your profile/dashboard of users on your app. You would also need to add the root URIs of your app that you will be using for testing or in production. For example, we can add `http://127.0.0.1:8000` to allow and use Auth0 in our development environment locally.  You also need to specify the callback URLs for your application which is `/complete/auth0` by default. ## Creating a BASH Script for integrating Auth0 So, we can now dive into creating the BASH Script to set up the Django x Auth0 application in minutes. The script is quite large, like 200 lines but don't worry! Its automation reduces the pain of integrating a User Authorization flawlessly. I am also thinking of adding the `cURL` command and parsing in the Client ids, keys, and secret keys, etc. ### Appending to a file We can use the `cat` command to append text to a file, using the syntax as below: ```shell cat << EOF >> filename text more text EOF ``` Remember here EOF is just a label to stop the command and save it to the file.  So, we can see that we were able to append to a file, multiple lines using the cat command. We have used this concept in adding configuration and credentials to the `settings.py` or the `urls.py` files. ### Adding text before a particular line using `sed` `sed` is a great command, and there is nothing you can't do with it, OK there might be exceptions. We can get write to a file directly (don't display the output) and specify the line number before which we want to append the text. We can then add the text we want and followed by the filename. ```shell sed -i '33 i sometext here' filename ``` Here, `33` is the line number in the file which we want to insert before. We have used `'"'` to add a `'` inside a `'`, this might feel a bit wired but that is how it is in BASH. Let's say you want to add `print('Hello, World!')` to a particular line, we have to enclose `'` with these `"'`( double and single quotes), ```shell sed -i '2i print('"'Hello, World'"')' hello.py ``` This will add the line `print('Hello World')` to the file `hello.py`  ### Appending to a line using sed We can even append text to a particular line using sed, we can use some escape characters and regex to add the text from the end of the line. ```shell sed -i '2i s/$/ textgoes here /' filename ``` Here 2 is any number of line you want to add text to, next `i` a prompt for inserting text and then we have regex like `s/$/ /`, this will put the text enclosed in `/ /` to the end of the line as indicated by `$`. We have the filename at its usual place as before. So, lets say, I want to add a comment to the second line in the previous example, I can use the following command to do it: ```shell sed -i '2 s/$/ # another comment/' hello.py ```  We have used these commands to add the `include` function in the `urls.py` in the project folder. So those were all the operations we used for doing some automated editing for the Auth0 app integration to our Django project. Below is the entire script and is also uploaded on [GitHub](https://github.com/Mr-Destructive/django-auth0-quick-setup). ## Complete BASH Script You can run the file by parsing the name of your project. ```shell bash script.sh mywebsite ``` Wait for some 2-3 minutes, and the script will produce the Django application with the Auth0 app integrated. You will have to enter the credentials manually wherever applicable. ```bash #!/usr/bin/env bash mkdir $1 cd $1 pip install virtualenv virtualenv venv source venv/Scripts/activate pip install django django-admin startproject $1 . cat << EOF >> requirements.txt social-auth-app-django~=3.1 python-jose~=3.0 python-dotenv~=0.9 EOF pip install -r requirements.txt pip freeze > requirements.txt python manage.py startapp auth0login touch auth0login/urls.py mkdir auth0login/templates touch auth0login/templates/index.html touch auth0login/templates/dashboard.html sed -i '40 i \ '"'"'social_django'"'"',' $1/settings.py sed -i '41 i \ '"'"'auth0login'"'"',' $1/settings.py sed -i '21 i \ path('"''"', include('"'auth0login.urls'"')),' $1/urls.py sed -i '17 s/$/, include/' $1/urls.py cat << EOF >> $1/settings.py SOCIAL_AUTH_TRAILING_SLASH = False # Remove trailing slash from routes SOCIAL_AUTH_AUTH0_DOMAIN = 'YOUR_DOMAIN' SOCIAL_AUTH_AUTH0_KEY = 'YOUR_CLIENT_ID' SOCIAL_AUTH_AUTH0_SECRET = 'YOUR_CLIENT_SECRET' EOF cat << EOF >> $1/settings.py SOCIAL_AUTH_AUTH0_SCOPE = [ 'openid', 'profile', 'email' ] EOF python manage.py migrate cat << EOF >>auth0login/auth0backend.py from urllib import request from jose import jwt from social_core.backends.oauth import BaseOAuth2 class Auth0(BaseOAuth2): """Auth0 OAuth authentication backend""" name = 'auth0' SCOPE_SEPARATOR = ' ' ACCESS_TOKEN_METHOD = 'POST' REDIRECT_STATE = False EXTRA_DATA = [ ('picture', 'picture'), ('email', 'email') ] def authorization_url(self): return 'https://' + self.setting('DOMAIN') + '/authorize' def access_token_url(self): return 'https://' + self.setting('DOMAIN') + '/oauth/token' def get_user_id(self, details, response): """Return current user id.""" return details['user_id'] def get_user_details(self, response): # Obtain JWT and the keys to validate the signature id_token = response.get('id_token') jwks = request.urlopen('https://' + self.setting('DOMAIN') + '/.well-known/jwks.json') issuer = 'https://' + self.setting('DOMAIN') + '/' audience = self.setting('KEY') # CLIENT_ID payload = jwt.decode(id_token, jwks.read(), algorithms=['RS256'], audience=audience, issuer=issuer) return {'username': payload['nickname'], 'first_name': payload['name'], 'picture': payload['picture'], 'user_id': payload['sub'], 'email': payload['email']} EOF cat << EOF >> $1/settings.py AUTHENTICATION_BACKENDS = { #'YOUR_DJANGO_APP_NAME.auth0backend.Auth0', 'django.contrib.auth.backends.ModelBackend' } EOF cat << EOF >> $1/settings.py LOGIN_URL = '/login/auth0' LOGIN_REDIRECT_URL = '/dashboard' EOF cat > auth0login/views.py<<EOF from django.shortcuts import render, redirect from django.contrib.auth.decorators import login_required from django.contrib.auth import logout as log_out from django.conf import settings from django.http import HttpResponseRedirect from urllib.parse import urlencode import json def index(request): user = request.user if user.is_authenticated: return redirect(dashboard) else: return render(request, 'index.html') @login_required def dashboard(request): user = request.user auth0user = user.social_auth.get(provider='auth0') userdata = { 'user_id': auth0user.uid, 'name': user.first_name, 'picture': auth0user.extra_data['picture'], 'email': auth0user.extra_data['email'], } return render(request, 'dashboard.html', { 'auth0User': auth0user, 'userdata': json.dumps(userdata, indent=4) }) def logout(request): log_out(request) return_to = urlencode({'returnTo': request.build_absolute_uri('/')}) logout_url = 'https://%s/v2/logout?client_id=%s&%s' % \ (settings.SOCIAL_AUTH_AUTH0_DOMAIN, settings.SOCIAL_AUTH_AUTH0_KEY, return_to) return HttpResponseRedirect(logout_url) EOF cat << EOF >> auth0login/templates/index.html <div class="login-box auth0-box before"> <img src="https://i.cloudup.com/StzWWrY34s.png" /> <h3>Auth0 Example</h3> <p>Zero friction identity infrastructure, built for developers</p> <a class="btn btn-primary btn-lg btn-login btn-block" href="#">Log In</a> </div> EOF cat << EOF >> auth0login/templates/dashboard.html <div class="logged-in-box auth0-box logged-in"> <h1 id="logo"><img src="//cdn.auth0.com/samples/auth0_logo_final_blue_RGB.png" /></h1> <img class="avatar" src="#"/> <h2>Welcome {{ user.username }}</h2> <pre>{{ userdata }}</pre> </div> EOF cat << EOF >> auth0login/urls.py from django.urls import path, include from . import views urlpatterns = [ path('', views.index), path('dashboard', views.dashboard), path('logout', views.logout), path('', include('django.contrib.auth.urls')), path('', include('social_django.urls')), ] EOF python manage.py makemigrations python manage.py migrate ``` ## Conclusion Ok, so this was it, a quite big script but that's how automation can be. We were able to set up a Django base application with a ready app of Auth0 to extend the functionality. This was just a basic script also you can extend the functionalities like adding a curl command to fetch the credentials and make it more automated but that was not the aim of this article. If you had any issues using the script please let me know, I'll be happy to fix those. Thanks for reading. Happy Coding :) https://www.meetgor.com/posts/oop-cpp-p3 Introduction After learning a bit about classes, objects, and access modifiers, we can move on to another important concept of Constructors and Destructors. Thi Thu, 23 Sep 2021 00:00:00 UTC ## Introduction After learning a bit about classes, objects, and access modifiers, we can move on to another important concept of Constructors and Destructors. This part will cover Constructors. We'll explore what are constructors, why we need them, types of constructors, and their semantics in C++ as usual. So without wasting any time, let's get started. ## What is a constructor A constructor is a method just like another method in a class, but it has a special name for a reason. The name itself speaks, `Constructor`, which will get called just after the object gets created. A constructor is just a method but it should only have the name same as of the class, that's the thing that distinguishes it from other methods. The main thing that stands out for a constructor is that you don't have to call it. It gets called automatically (just magically), yes you don't have to call it explicitly. That's a great feature in Object-Oriented Programming. We can control the flow of the program according to our needs. A constructor has certain features or principle that makes it different from the rest of the class methods: - There is no return type for a constructor. - The name of the constructor should be the same as the class name. - There can be multiple constructors for a class (depending on the parameters passed to it). - A constructor gets called automatically when an object is instantiated (created). ## Why do we need a constructor? Let's suppose you wanted to initialize some variables (properties) before the main loop of a program starts or initiates. This can be done manually by accessing them in the mail function but for many objects, this can get really boring to write every time. This is where the constructor comes in and just improves the overflow structure and manageability of the program.  This looks silly but is actually used a lot just beyond initialization, you might want certain things to be done as soon as the object gets created. All ideas and flow of the program in a particular desired manner are all possible due to constructors. It is also a way of automating several repetitive tasks for better design and maintaining the codebase. ## Define a constructor for a class in C++ Now, it's time to get into code, and let's see how to define a constructor in a class using C++. ```cpp #include "iostream" using namespace std; class blog { public: // defining a constructor for the blog class blog() { cout<<"Call from constructor "; } }; int main(){ blog post1; return 0; } ```  This is how you define a class in C++. A constructor doesn't even have a return type of void, literally, it doesn't return anything. A Constructor can do anything another normal method can do except to return anything. We can also pass certain parameters to it as well. The thing here is that, the constructor should be public if you want to call it from the main function or anywhere else outside the class(class itself / friend class / derived class). You can make it private or protected as per your needs and the hierarchy of your application. If you want to know more about those access modifiers(public/private/protected), you can check the previous part of this series. > Remember, you cannot call the constructor from the main function if it is not public. We can actually create some good examples where the constructor plays a vital role in the design and the feasibility of the actual idea. Let's say we want to create three objects, each taking the same input of properties from the user but the value will be different as obvious and also determine another property based on an input. Should we write those inputs for every object or write a function for the same. The latter will be the best choice, but which function are you thinking of? If that's a constructor then you are correct! Ya we can even write a normal method but we have a print function called up already, that might create some undesired behavior like unreferenced variables and segmentation fault while dealing with complex data structures or large applications. ```cpp #include "iostream" #include "string" using namespace std; class Animal { public: int legs; string name; string type; Animal() { cout<<"Who is the Animal? "; cin>>type; cout<<"Enter the name of Animal? "; cin>>name; cout<<endl; if(type.compare("dog")==0) legs=4; else if(type.compare("monkey")==0) legs=2; else legs=0; } void print() { if(legs==0) cout<<type<<"'s name is "<<name<<endl; else cout<<name<<" has "<<legs<<" legs. "; cout<<endl; } }; int main(){ Animal obj1,obj2,obj3; obj1.print(); obj2.print(); obj3.print(); return 0; } ```  We can see how well this can scale for creating many objects. This is just one of the examples where the constructor just single-handedly takes care of many things. We can always create an array of objects but to keep things simple here, we have hardcoded the object names. ## Types of Constructors Well, won't it be nice to pass the values to the constructor and then do certain computations? There is a solution to that, we can overload functions i.e we can create multiple constructors each having a unique combination of parameters associated with it. The following are the types of constructors: 1. Default Constructors 2. Copy Constructors 3. Parameterized Constructor ### Default Constructor This is the constructor which we have used so far, even if we don't define a constructor, the C++ compiler automatically creates one which does nothing. We have already seen the normal constructors without any parameters, just a simple function that does some I/O to the console. ### Copy Constructor This is a great feature of constructors, you can copy certain properties of previously created objects or define new ones and pass them to the constructor. This is why I said we can have multiple constructors depending on the number of parameters or the type of parameters passes to it. So, with that said, we can basically define constructors for different purposes with different requirements and conditions. This just opens up a new dimension of creating complex functions and programs with great flexibility and adaptivity. ```cpp #include "iostream" #include "string" using namespace std; class Animal { public: int legs; string name; string type; Animal() { cout<<"Who is the Animal? "; cin>>type; cout<<"Enter the name of Animal? "; cin>>name; cout<<endl; if(type.compare("dog")==0) legs=4; else if(type.compare("monkey")==0) legs=2; else legs=0; } // copy constructor with reference of an object of the same class Animal(Animal &obj) { cout<<"This is my pet "<<obj.type<<", "<<obj.name<<endl; } void print() { if(legs==0) cout<<type<<"'s name is "<<name<<endl; else cout<<name<<" has "<<legs<<" legs. "; cout<<endl; } }; int main(){ Animal obj1; Animal obj4(obj1); obj1.print(); return 0; } ```  In the above example, we have defined another constructor that accepts a parameter that is a reference to another created object and basically does some I/O operations. Why would we need this? Well, the name itself says it all. `To copy the values of existing objects` with slight or minor modifications. It basically depends on how you want to copy the object into another. This can also be applied to additional class properties and just change the copy constructor and there it is, big tasks performed with minimal changes. We can also be used to modify the existing objects (though it can get a security issue, still pretty cool) by accessing the properties of that object in the copy constructor. ### Parameterized Constructor Now, we know that we can pass parameters to our constructors, let's exploit that functionality to create all sorts of constructors. No, I'm just kidding, you can create multiple constructors by passing in various combinations of parameters to it. Let's take an example, instead of taking input from the default constructor we can pass in the values while creating the object, and the constructor which matches the parameters gets called. ```cpp #include "iostream" #include "string" using namespace std; class Animal { public: int legs; string name; string type; // default constructor Animal() { cout<<"Who is the Animal? "; cin>>type; cout<<"Enter the name of Animal? "; cin>>name; cout<<endl; if(type.compare("dog")==0) legs=4; else if(type.compare("monkey")==0) legs=2; else legs=0; } // constructor with two parameters (both String) Animal(string type, string name) { cout<<"This is my pet "<<type<<", "<<name<<endl; } // constructor with one parameter(String) Animal(string name) { cout<<"Hello, "<<name<<endl; } void print() { if(legs==0) cout<<type<<"'s name is "<<name<<endl; else cout<<name<<" has "<<legs<<" legs. "; cout<<endl; } }; int main(){ Animal obj1; Animal obj2("dog", "scooby"); Animal obj3("Pomello"); obj1.print(); return 0; } ```  We created the first object without any specification but the second object was passed with two parameters and we indeed called the correct constructor. The third object was just given a single parameter and again called the required constructor. So, we can see that a particular constructor is called which matches the parameters passed to it. That's quite an intelligent and smart design. This just improves the overall diversity and adaptability hand in hand. #### Function/Constructor Overloading > When a single function(same name) has different implementations depending on the parameters passed to it is called Function overloading. The thing above i.e Parameterized Constructors is technically called Function Overloading (Constructor in this case). We will look into this topic in much detail when we cover more on methods. But this might be enough to get your head around the concept of overloading in OOP. Let's say we have a function that takes two parameters by default but also can take three parameters when given. We can process them by having a different implementation. This creates a lot of freedom and broadens the scope of customization in complex applications. ```cpp #include "iostream" using namespace std; void intro(string name, int age) { cout<<name<<" is "<<age<<" years old. "; } void intro(string name) { cout<<"Hello, "<<name<<endl; } int main(){ intro("Ash",10); intro("Brock"); return 0; } ```  In this example, we have a single function `intro` but it has two implementations, one has a two-parameter and the other has just one. This is called Function Overloading. We just applied this in the functions in classes called constructors but the concept is the same. We'll dive into Overloading in the next few sections. ## Conclusion That is it from this part. We'll see `Destructors` in the next one. From this part, we learnt about constructors and how important they are in creating the desired flow to our application. We also touched on the types of constructors and function overloading a bit, which are also important aspects of OOP. That's it from this one. Happy Coding :) https://www.meetgor.com/posts/techstructive-blog-init Introduction **Hello World!** This is an article in which I'll introduce why Techstructive Blog exists and what it covers in brief. This is just a casual introd Sun, 19 Sep 2021 00:00:00 UTC ## Introduction **Hello World!** This is an article in which I'll introduce why Techstructive Blog exists and what it covers in brief. This is just a casual introduction to my blogging niche. P.S. It's not any technical blog but has challenged me enough to write and complete it. > Techstructive Blog is my personal blog hosted on GitHub pages as well as on [Hashnode](https://techstructiveblog.hashnode.dev/). It first started on Hashnode and soon created my own using Jekyll on GitHub pages. On this blog, I write articles mainly Vim, BASH, and some programming languages and tools. Firstly thanks to Hashnode for providing a free platform for developers to share their journey to the world. I am happy that I have found such a gold mine of a supportive and growing community of developers that help find and inspire each other's way. ## What Techstructive Blog covers? > Vim + Linux + Random Stuff Techstructive Blog started around 20th February 2021, where I first found Hashnode. There I wrote my first article in my technical writing journey. The article was about `Vim`. It covered some quite basic stuff on how to get started in Vim. After few months I really found the need to blog about my learnings as I felt more confident about my skills after writing 10-12 articles on the [GeeksforGeeks portal](https://auth.geeksforgeeks.org/user/meetgor/articles). I again started writing articles about Vim and BASH as well. Fast forward today, [Techstructive Blog](https://techstructiveblog.hashnode.dev/) has 26 articles on it, 14 being about Vim and 8 being about BASH. That is the niche of Techstructive Blog revolving around GNU\Linux and its applications. I also talk about Django and C++ as they are technologies I love to work with and create some personal projects with them. ## Rambling about Vim I don't know it might feel to the readers like I am rambling and boasting about how much I know about Vim, but I honestly love and adore the text editor as it is so customizable and wicked fast. I just write about what might be helpful and important to others as a setup or just editing in Vim. I personally use Vim every day so I feel more confident and excited to write about it. I feel it's just the start of Vim for me, I can go on writing articles about Vim that will be helpful for someone or the other. I have 7 articles already planned up for Vim, I just need to finish off certain pending tasks and projects first. I am gonna primarily focus on how to use Vim for various languages or frameworks. This also applies to using Vim in different environments and improving the workflow of daily usage. Also notice that all my Vim articles are short and simple. They just cover the topics as mentioned in the title, because I don't feel like spitting everything in one article that no one can understand and connect topics together. Instead, I break topics into series, just like the Vim: [Tabs](https://techstructiveblog.hashnode.dev/vim-tabs), [ Window Splits](https://techstructiveblog.hashnode.dev/vim-window-splits), and [Buffers](https://techstructiveblog.hashnode.dev/vim-buffers) articles. I plan ahead as more and more articles about Vim and maybe some weekly newsletter of Vim tricks and commands. That's some exaggerated thing but I have decent skills to tackle that. Who knows what the future holds? but it's important to plan your way. ## Linux Well, I technically don't have any articles about `Linux` but I have a lot of things to share about it. I have a series to be written about it with a minimum of 10 articles in it. But I am not able to start my way off in search of a perfect time. Though I have some BASH articles and a [series](https://techstructiveblog.hashnode.dev/series/bash-scripting) about scripting already. I have always preferred to use Linux even if I use Windows 7 daily, I have a virtual machine to satisfy some of my needs and curiosity. That's aside I have used Termux which is a terminal emulator for Android, I have literally spent the Lockdown, coding on it all the months and weeks. I feel to talk about the need to use Linux and how it has revolutionized programming and development around the world. This is really a biased topic because people who use certain OS are gonna keep using it but I hope that they would get some insights at least from those articles. ## Web development I have created several projects/websites using Vanilla Javascript and Django, so I can pretty much talk about the things which I've learned and the mistakes that have made. I have a good grasp on Django now so I feel much confident to write series about it, just covering the concepts and the terminologies in which I faced challenges in getting through. ## General Programming Well, I can't lie but I feel too excited when it comes to learning new things and I genuinely learn them until I create something from it, may it be C/C++, Python. Django, Javascript, BASH, or other tools or frameworks. I am not limiting myself to any language or topic, but ya I can proficiently write about one and slowly gather around others as well. > Jack of all trades, master of none. That might be a good quote applicable to me but I believe in [PI-Shaped Learning](https://appunite.com/blog/become-a-pi-shaped-developer). PI-Shaped learning emphasizes broadening the knowledge in all domains but having expertise in one. Like I can talk about Web development, Android development, DevOps but I will be majorly inclined towards one and have a deep knowledge of ins and outs of that domain. ## Conclusion Phew! That took a while to finish. But hey! Thanks a lot for reading till here. I hope you got my thoughts about the scope of this blog. I owe to continue this blogging journey in the future and improve my as well as the reader's skills and experience. Well, until then HAPPY CODING :) https://www.meetgor.com/posts/oop-cpp-p2 Introduction Moving on to Part 2 of Object-Oriented Programming in C++, we will look into the concept of access modifiers which we skipped in the first section. Sat, 18 Sep 2021 00:00:00 UTC ## Introduction Moving on to Part 2 of Object-Oriented Programming in C++, we will look into the concept of access modifiers which we skipped in the first section. This won't be a comprehensive guide on access modifiers as it requires some other concepts like `Inheritance` which we will introduce in the next few sections. This series will be like connecting the pieces of OOP together and building a good foundation. In this part, we will discuss the access modifiers and their related concepts. The below is a gist of what will be covered in this part. - Understanding Access Modifiers - Type of Access Modifiers 1. Private - Getters - Setters 2. Public 3. Protected - Introduction to Derived and Friend classes Without wasting any time, let's roll in. ## What are Access Modifiers Access Modifiers as the name suggests they are used to change the accessibility of properties and methods in a Class. That means if we want a certain scope of our code to access the class members we can specify that using `access modifiers`. Access Modifiers form the basics or the pillar for `Data Encapsulation` in OOP. ### Data Encapsulation Data Encapsulation is terminology in OOP to keep all the components in a single entity or unit. Here the components are `properties` and `methods` and they are contained in a single unit called `Class`. We need to carefully use the class members i.e. properties and methods in order to avoid security and limiting certain members to be available for a specific scope or block of code. Hence, the Encapsulation of class members helps in structuring the class in a secure and allows access in a particular expected way. ## Types of Access Modifiers in C++ We do have certain modifiers or limiters for accessing class members in C++. We can either keep the class members limited to the class itself (and friend class), keep them open to the global scope, or restrict them to certain classes only(derived and friend classes). Let's discuss them one by one. - ### Private This access modifier allows us to access the class members only to the class methods and the friend functions and classes. We will discuss what friend classes are in the next few parts of the series. This restricts the usage of the class members directly from the main function or other scopes in our program. You can use `private:` to indicate the below declared properties and methods are private to the class. ```cpp class Name { private: int name; //other properties and methods } ``` OR (don't specify anything it **is private by default**) ```cpp class Name { int name; //other properties and methods } ``` This is assigned to every member of the class by default. So that is why we explicitly told to make it public in an example in the previous part of the series. Let's see what happens if we do not make it public. ```cpp #include<iostream> using namespace std; class Animal{ int legs; string name; void print() { cout<<name<<" has "<<legs<<" legs. "; } }; int main() { Animal dog; dog.name="Floyd"; dog.legs=4; dog.print(); return 0; } ```  You can see we cannot directly use those properties and methods which are set as private. It is a convention to keep properties private and create public methods to access those from the rest of the program. The public methods used to access and modify the value of the private properties are called `getters` and `setters` respectively. #### Getters Getters are the public methods of a class that are used to access a value to the private properties of that class. It is a function that returns the data of that particular property. We access the function as normally as we access the public functions. We define the getter function as: ```cpp datatype getpropertyname() { return propertyname; } ``` We can access the getter function as a normal public function, but this function returns a value, so we can store it in a variable and do all sorts of things. ```cpp classname objname; court<<objname.getpropertyname()<<endl; ``` #### Setters Setters are the public methods of a class that are used to assign/modify the value of the private properties of that class. It is a function that simply assigns the private properties to data of the data which is passed in as an argument to the function. We access the function as normally as we access the public functions. We have to pass in the value to assign the property to the function. We define the setter function as: ```cpp datatype setpropertyname(datatype x) { propertyname = x; } ``` We can access the setter function as a normal public function, but this function takes in a parameter, so need to pass in the value to assign it the same as the correspondent data type of that property. ```cpp classname objname; objname.setpropertyname(data); ``` So, after applying the getter and setter concept to our example, we can use them and make the properties private without any issues. ```cpp #include<iostream> using namespace std; class Animal{ int legs; string name; public: // take in a parameter of type same as of the property void setName(string dogname) { name=dogname; // assign the property a value same as of the parameter } // return type should be same as of the property string getName() { return name; } void setleg(int noflegs) { legs=noflegs; } int getleg() { return legs; } }; int main() { Animal dog; dog.setName("Flyod"); dog.setleg(4); cout<<dog.getName()<<endl; cout<<dog.getleg()<<endl; return 0; } ```   The above code runs successfully, and hence we make our program more secure and provide limited access to the rest of the components. You can see how we have passed the data as the parameter to the setter function, it should be the appropriate data-type with the property you are trying to set and also the same return type for the getter function. The code is also much readable and structured for others to read and understand. It might seem a silly thing but it really improves the maintainability in the longer run. - ### Public This might be familiar till now, we have been using this access modifier till now and it is quite straightforward to understand. We have created the getter and setter function using this access modifier. So, the `public` access modifier makes the class members be accessible anywhere in the program. This might be OK for many small applications but it is not ideal to use them for production-level applications as it might cause undesired consequences i.e. BUGS. We need to explicitly write public in the class definition, as `private` is set by default. So, as a simple example, as we saw in the previous part, it can be a lot easier to go with the public to understand OOP but the main OOP really shines in aspects like Encapsulation and Inheritance of Classes, which we will surely see in the upcoming parts. This is the example from the previous part: ```cpp #include<iostream> using namespace std; class Animal { public: int legs; string name; void print() { cout<<name<<" has "<<legs<<" legs. "; } }; int main() { Animal dog; // create an object of class Animal dog.name="Floyd"; // assign the class property to actual data in memory dog.legs=4; dog.print(); // call a method(function) associated to object's class return 0; } ```   As explained earlier, it is accessible to the main function or other scopes as well. We can access them using the dot separator(`.`) to assign it or to call the method. - ### Protected This is quite a handy access modifier, as it allows us to have the flexibility to keep the class members private and accessible to its derived or friend class. We will see the derived and friend classes in the next few parts. But for now, let's understand in an abstracted way. #### Derived Class(Child Class/ Sub Class) This is a concept in Inheritance, i.e. to pass the members of a class to another. So, there has to be two classes, the main(parent class) and another class that will inherit or take in the members from the parent class. So, the derived class has access to its public and protected members only. #### Friend class A friend class is a class that is allowed to access its **private** and protected properties or methods. It is kind of a special tag assigned to a class that it can access certain class' members. We will see it in detail afterward, right now, it's enough to understand, friend class is a class that can access a particular class' members may it be private or protected. The difference between a friend and a derived class is that a friend class can access the private members of the class to which it is a friend, but a derived class can't. Friend class also can't be inherited. Again, we will see this is in detail. So, it doesn't make sense for me to explain protected here. But just assume friend classes and derived classes are a thing. ```cpp #include<iostream> using namespace std; class Animal { public: int legs; string type; void print() { cout<<type<<" has "<<legs<<" legs. "; } protected: string name; }; int main() { Animal dog; dog.type="dog"; dog.legs=4; dog.name="Floyd"; dog.print(); return 0; } ```  This gives an error as protected members can be accessed only by derived or friend classes since we are accessing it from the main function, it's not allowed in the global scope. We can only access the protected members from the derived or friend classes. So, let's derive a class from the base class(Animal), and after that, we can see protected members in action. ```cpp #include<iostream> using namespace std; class Animal { public: int legs; string type; void print() { cout<<type<<" has "<<legs<<" legs. "; } protected: string name; }; class Pet:public Animal { public: void print() { name="Floyd"; cout<<name<<" is a "<<type<<endl; cout<<type<<" has "<<legs<<" legs. "; } }; int main() { Pet dog; dog.type="dog"; dog.legs=4; dog.print(); return 0; } ```   We have accessed the protected property `name` in the derived class `Pet`. Yes, this is how we derive a class in C++, ```cpp // A child class serived from base class class NewClassName: public BaseClassName { // properties and methods }; ``` So, after deriving the `Pet` class from `Animal` class, we have access to its public and protected class as well. We simply assigned one of the protected members from its base class i.e. `name` and used it in the function `print`. This is how you can use protected members in a derived class, also you can do it in friend class, but we will look at that in a separate part. Remember you can **use getters and setters as well to assign and access those protected members**. Keeping it simple and easy to understand is what a protected access modifier can provide. **Protected access modifiers are quite great and provide much more flexibility than `private` and more privacy than `public` access modifiers.** ## Which to use when? It is often a good practice to use private members, but it might be not possible to make everything private, so we can use certain properties as private, some methods as public and protected as well as per requirement and complexity.  Looking at the above chart, you can analyze your problem and work a way around to find the perfect secure match for your application. Remember to use public members only when you have to explicitly use through the program. If you find that, this member shouldn't be introduced in certain scope then use make it private or protected. If there is some kind of hierarchy in the program and its components, analyze and make a structure of it to have a better understanding of the program flow. ## Conclusion So, from this part, we were able to understand the access modifiers and got somewhat of a dive into Inheritance. There were some concepts like Friend and Derived classes, Inheritance, Encapsulation which were just explained in short, but they are a topic that deserves separate attention. We'll discuss them in the upcoming parts. Thanks for reading. Hope you understood the concepts in a better way. Until then, as always, Happy Coding :) https://www.meetgor.com/posts/oop-cpp-p1 Introduction We all know that C++ is famously known for Object-Oriented Programming, but what is Object-Oriented Programming? Well, this is the article, where w Thu, 16 Sep 2021 00:00:00 UTC ## Introduction We all know that C++ is famously known for Object-Oriented Programming, but what is Object-Oriented Programming? Well, this is the article, where we will explore the basics of Object-Oriented Programming, its building blocks i.e Classes and Objects, and get familiar with its basic semantics in C++. Let's get into it straight away! ## What is Object-Oriented Programming? Object-Oriented Programming(OOP) is a way to structure and design a program or an application. Object-Oriented Programming deals with creating classes and objects and it's related terminologies. OOP helps in following the principles like DRY(Don't Repeat Yourself) and KISS(Keep it Simple Stupid). Using OOP certain applications and problems can be solved in a simple and efficient way. It helps in better understanding and readability of the code. > Virus is a class and COIVD-19 is an object of the Virus class XD In OOP we create a collection of the data and functionalities by organizing in a structure called `Class`. We then instantiate or create the actual data by creating an `object` of the particular class. Once we have created a class, we can simply create as many objects as we need to. Objects are basically the storage of the actual data in memory. Let's explore them in the next few sections. NOTE: There are a lot of terminologies in OOP, just remember the basics of those terms and not the different names of those terms. ## What are Classes Classes are the structure or template of the data and its associated behavior. The data or the variables in the classes are called `properties` or `attributes`(also `data members`). We also have functions that define the behavior of the properties or the data present in the class and are called `methods`. A Class in C++ should be defined outside of the main function. In C++, `class` is a reserved word and hence it is used to declare and define a class. It has a general structure like: ```cpp class Name { // properties and methods }; ``` It's a convention to use the initial letter of a Class Name Uppercased. We can define class methods outside the class body as well, we will see it in the next few parts of the series. Let's create a basic class in C++, ```cpp class Animal { public: int legs; string name; void print() { cout<<name<<" has "<<legs<<" legs. "; } }; ``` Here in the above code, we have a class called `Animal` and it has 2 properties called `legs` which is an int and `name` a string. We also have a method called `print` which displays the current object's name and the leg count. We will see what is the current object in the next section.  We use those properties or the variables defined in the class in our application either by accessing them through the objects or in the methods of that class. The variables or properties in classes are not assigned to any memory address i.e. they are just kind of a blueprint or placeholder tags to match for the data. ### Access Specifier We have written `public` before the definition of the properties of the class, it is an access modifier. Don't worry we will see it in detail in the next few parts of the series, just for time being understand that we can change whether we want the main function(or any function globally) to access the class' properties or not. We are saying `public` which means we can access the properties of this class anywhere outside the class. This is not a good practice but just for simplicity of understanding, we are using public. **If you do not specify anything like `public`, it is by default `private`, which means the properties of the class are not accessed anywhere outside the class**. Let keep it simple right now. We also have other concepts like defining the methods outside/inside the class, header files, constructors, destructors, and many others related to a class definition, we will cover it in the next few parts. ## What are Objects Now, we have created a class but where is the data actually? It will be in `objects`, We actually assign the data or properties to a memory address by creating the objects of that particular class. We can create objects of a particular class in C++ by writing the name of the class that this object will belong to and the name of the object,`classname objectname;`. This will create or assign memory to the properties of the class to the object. After the object has been created, we can assign the value to the properties of the class in the object. We can access the properties of the class by referencing the name of the object with `.` and the name of the property or variable of the class, `objectname.propertyname`, we can assign the value simply by using the assignment operator `=` and give the properties the value, input from the user or however you like it. ```cpp #include<iostream> using namespace std; class Animal { public: int legs; string name; void print() { cout<<name<<" has "<<legs<<" legs. "; } }; int main(){ Animal dog; // create an object of class Animal dog.name="Floyd"; // assign the class property to actual data in memory dog.legs=4; dog.print(); // call a method(function) associated to object's class return 0; } ```  If Animal is the class, then the dog here is the object. As simple as you can think. We are technically **instantiating an object** when we say `Animal dog`. We assign the variables the memory as defined in the class. We give some value to the properties using the `.`(dot syntax), after the class name. So `dog.legs = 4;` will assign the value of 4 in the memory address of the object's property(dog is the class and legs is the property). Similarly, we can call the function associated with the class as we do for properties but we use `()` to indicate to calling the function.  So that is how we create objects in C++. ## Why and When to use OOP Object-Oriented programming makes quite complex problems and applications structure and scale quite easily and enhances the readability of the codebase. We can use OOP to create the applications for bigger and real-life applications, it allows us to add new features quite easily and thereby improving the maintaining ability. The following might be the reasons to use OOP in an application: - If the application cannot be stated in a single sentence, you need to create the components into classes and hence use OOP. - It is quite easy to maintain OOP applications even if there are a lot of maintainers to the project. - If you see certain parts of the code repeating in many places, OOP is the way to go. - If you want your application to be secure in terms of data from the rest of the components, OOP is again a great option as you can use Encapsulation to the advantage. ## Conclusion So, we have studied the concept of Object-Oriented Programming and how to define classes and create objects in C++. There are a lot of concepts in OOP to grab up in the next few parts of the series, but this should build the foundation of OOP as a paradigm or a way of thinking when it comes to problem-solving and programming. I hope you have got a good introduction to the basics of Object-Oriented Programming and its semantics in C++. Thank you for reading. In the next part, we will explore the Access Modifiers/Specifiers in the Classes. Until then Happy Coding :) https://www.meetgor.com/posts/technical-writer-journey Introduction Hello World! I am Meet Gor, a self-taught web developer, Linux enthusiast, and a newbie technical writer. I am currently studying in an Engineering Tue, 14 Sep 2021 00:00:00 UTC ## Introduction Hello World! I am Meet Gor, a self-taught web developer, Linux enthusiast, and a newbie technical writer. I am currently studying in an Engineering college pursuing my B. Tech in I.T. I love to explore new programming languages and technologies, create innovative projects, love to write stories and blogs like this. ## How it started When I was learning Vim, I found the need to write some kind of cheatsheet for myself. So I started writing it, soon I realized I had written a kind of an article instead of a cheatsheet XD There were a lot of chunks of it unorganized but meaningful in some sense. So, from there I started to research on article writing. I found the term perfect for my techno feelings to express i.e. "Technical Writing". I soon landed on Hashnode, I knew Medium, GeeksforGeeks but found Hashnode absolutely perfect and much more viable to the creativity of an author. ### Starting on Hashnode My [first article](https://techstructiveblog.hashnode.dev/why-and-how-you-should-use-vim-as-a-text-editor-and-ide) was published on [Hashnode](https://hashnode.com/@Mr-Destructive/joinme), which I absolutely love and adore. That was a bit big article about 12 minutes for a beginner me writing everything about Vim which I knew about that time. The response was not that bad, I received a couple of reactions and comments to pump me up but I never felt confident to write more articles. ### GeeksforGeeks Internship Until I found an **Internship at GeeksforGeeks for a Technical Writer** role. [Geeksforgeeks](https://www.google.com/aclk?sa=l&ai=DChcSEwj-_8TAq_7yAhUtlUsFHdI4AD4YABAAGgJzZg&ae=2&sig=AOD64_1xL_mUI1OLWumXOK-H1jF4wRP8eg&q&nis=1&adurl&ved=2ahUKEwjSlb_Aq_7yAhXMb30KHXM6BPsQ0Qx6BAgCEAE) is a start-up for Computer science students to learn and grow with free as well as paid courses and mentors. In a couple of months, I wrote some 10-12 articles about Shell scripting. There I really found my interest and the confidence I wanted. It was really a great experience working here and loved the freedom it provided to the authors with genuine and constructive feedback along with a decent stipend. [Here](https://auth.geeksforgeeks.org/user/meetgor/articles) you can find my work. ### The start of the `Techstructive Blog` Soon, on June 6th something stuck in my mind that I should write about how to use Vim for doing various things. So, the first idea was what currently I was learning in College i.e Python, I started writing about the Vim set up for python and I completed it within 2 hours. I published it on Hashnode and **from there I never looked back. I started writing an article every 3-4 days**. That summed up for 3 months making 21 articles till September. I also made my personal open-source blog called [`Techstructive Blog`](https://mr-destructive.github.io/techstructive-blog/) using Jekyll and hosted it on GitHub pages.  I did experiments with a lot of blogging tools and frameworks like [Netlify with Flask](https://www.patricksoftwareblog.com/flask-tutorial/), [Django distill](https://github.com/meeb/django-distill), [Gatsby Js](https://www.gatsbyjs.com/starters/gatsbyjs/gatsby-starter-blog), etc and finally landed on [Github pages](https://pages.github.com/) with [Jekyll](https://jekyllrb.com/) as it is quite customizable and a lot easier to add certain components as well. ## Why I kept going? It's not hard to keep up with what you love to do. Even if you are not that interested in doing something, just having curiosity and consistency can turn any work into a skill. Here are the reasons why I love to keep going as a Technical writer ( some of them are from the Bootcamp session from @[Edidiong Asikpo](https://twitter.com/didicodes) and @[Quincy Larson](https://twitter.com/quincy) :) - Over a span of 5-6 months I have learned a lot than I had in my past 2 years, I understood that I am writing for myself first and not only for other people. - It helps me in improving my online presence and clearly showcases my skills and interests. - Able to sum up ideas and document them well. - Maintain a record of learning for your future self. - Learning to get things done by writing articles consistently in an allotted time. ## How I kept going? I am writing articles almost daily one for my blog and another day for GeeksforGeeks. I have a schedule and follow it almost punctually. I have never run out of ideas and motivation till now, every time I feel that there's no motivation in me, something or the other pops up just like this [Hashnode Bootcamp - The Art & business of Technical writing](https://hashnode.com/bootcamp). Some youtube videos, Hackathon, Bootcamps, Project, conversations with friends just can impact in a constructive and a drastic way, they create a refreshed experience and allow you to clarify the doubt in yourself. Just you have to be open to ideas and feedback from other people and learn in harmony. ## My writing workflow This is really a neglected part when you ask a technical writer as it can really improve your productivity. I use Hashnode and Vim both for writing out the markdown script. I like to write the subheadings or the content which I want to explain in that article. That really makes a bigger idea in a small chunk and I repeat the process until I feel that is the lowest abstracted way I can explain something. I record some screencasts in gifs on ScreenToGif recorder, MS Paint and Canva as an Image editor, **Hashnode makes uploading and accessing images, gifs so easy and it's quick too, really Hashnode is just flawless for a tech writer.** ## Conclusion That's been my story so far. I have grown from a noob coder to a somewhat confident nerd who can quickly write functional scripts, jump in Vim with more speed, and of course, teach someone something that can be helpful. It also is a way for me to relax and enjoy the process of programming. Thanks a lot if you have been reading till here. **Thank you Hashnode for providing such a great opportunity to create a side hustle here**. As always Happy Coding :) https://www.meetgor.com/posts/vim-for-cp Introduction Vim is not a bad text editor when it comes to using it for Competitive Programming. It's kind of one way or the other, you would love it or you cou Mon, 13 Sep 2021 00:00:00 UTC ## Introduction Vim is not a bad text editor when it comes to using it for Competitive Programming. It's kind of one way or the other, you would love it or you could trash it as it can waste a tremendous amount of time to write code every time. But once you are in a decent setup including some key-bindings and plugins can improve your speed in using Vim. I personally have used it since the beginning and haven't regretted it even a tiny bit till date. It's a flawless and enjoyable experience. I've used C++ for my CP journey, C is quite similar as well, but C++ has more library support and is a bit easier to write comparatively. Using Vim for CP is a bit challenging in the initial stage but just stay consistent and you'll be in flying colors on the other side. Here, I'll give some quick key-bindings and plugins to include to Vim to enhance the workflow for Competitive Programming. ## Quick Boilerplate code Firstly, let's create a key-binding for the boilerplate code. We just need to map the key with the code we need to write which will be integrated with Vim commands. For C++ ```vim nnoremap cpf i#include<iostream><Esc>ousing namespace std;<Esc>o<CR>int main(){<Esc>o<Esc>oreturn 0;<Esc>o}<Esc>kki ``` For C: ```vim nnoremap cp i#include<stdio.h><Esc>o<CR>int main(void){<Esc>o<Esc>oreturn 0;<Esc>o}<Esc>kki ``` For Java: ```vim nnoremap <C-j> iclass <ESC>"%pxxxxxa {<ESC>opublic static void main(String args[]){<Esc>o<Esc>o}<Esc>o}<Esc>kki<Tab><Tab> ``` Now, you get the idea of creating such macros for your own programming language. I don't think Python can have boilerplate code, even it has it's quite small compared to C/C++/Java/etc. You can include those codes into the key map. Let me explain the map to you. Let's look at the Java example, the `nnoremap` is the map command with attributes like non-recursive and the mode of mapping. In this case `n` stands for normal mode map i.e. the map will get triggered in the normal mode, `nore` stands for non-recursive behavior of the map. For further readings, you can read my [Vim-keymapping Guide](https://mr-destructive.github.io/techstructive-blog/vim/2021/06/14/Vim-Keymapping.html) article. The map itself has the right and a left part, kind of a map of the key and command to execute. You can use any key to trigger it, I have used CTRL+J or `<C-j>` to activate the map. The command is quite long but let's break it down into smaller chunks. - Starting with `i`, we enter into insert mode and we type in class and hit `<ESC>` key and this is a special trick to deal with java for the class name as it should be the name of the file itself. We use the `%` register for copying the current file's entire name (with extension), so we have to remove the `.java` part so, you see five x's to delete those after we have pressed a to get into insert mode just ahead of the cursor. We then insert `{` for the class and again hit escape to go back to normal mode. - We type `o` to go into insert mode to the next line, type in `public static void main(String args[])` and this is the same stuff of escaping and going in next line. - Finally, we hit `kk` to move up twice and enter the insert mode with `i` and hit tab twice to get the indentation correct. So, you can add more as per your preference of pre-defined constants and functions and the rest stuff involved in your language of choice. ## Running Code by a Key-binding After the boilerplate code being taken care of, we can now move to the building and running of the code file in Vim. We can add some more custom key bindings to auto compile and run the code within Vim. We can compile the code with the compilers or interpreters for the languages from the Vim using `!` which executes the commands from the terminal and pass in `%` i.e. the current file name and output file as `%:r` i.e. current file name without the extension in case of C/C++/Java. **Compile -> `c++ hello.cpp -o hello`** **Run -> `%:r` for Windows** and **`./%:r` for Linux/macOS** We have added those two commands i.e (compile and run) using `&&` to make things more clear and compact. For C++ ```vim nnoremap cpp :!c++ % -o %:r && %:r<CR> ``` For C ```vim nnoremap c, :!gcc % -o %:r && %:r<CR> ``` For Java ```vim nnoremap ,j :!javac % && java %:r<CR> ``` For Python ```vim nnoremap py :python %<CR> ``` Again, you can apply this to any programming language you use. Also kindly note that their could be some differences for compiling the code in Linux and Windows so just be careful which command you use in which Operating system.  The above gif demonstrates the usage of those commands and key bindings to make the workflow of running and building process much easier and smoother. ## Opening Terminal to compile and run code We can use the terminal to compile and run code if you wish to do it this way as well. Vim is quite a terminal-friendly editor, you can open any terminal from Vim. You can enter `:term` to open the default terminal in a horizontal split. You can explicitly enter the terminal/shell name to open it like `:term bash` to open bash, `:term powershell` to open PowerShell, `:term zsh` to open zsh, and so on. This is quite a great utility that vim provides with it. You can switch between windows using `<C-w>` or CTRL+W **twice** or use HJKL keys along with `<C-w>` to move with ease. More on Vim and Integrated Terminals in [this article](https://mr-destructive.github.io/techstructive-blog/vim/2021/06/29/Vim-Terminal.html). You can now explore the terminal world yourself and execute the commands from therein. ## Plugins When it comes to Vim, there are a ton of plugins made for everything you can imagine and couldn't imagine. We can use Plugin managers to manage those plugins for us or manually manage those yourself. You can go with one of the three well-known plugin managers like `Vim-Plug`, `Vundle` or `Pathogen`. Using those plugin managers, we can install plugins like some of the useful ones are: ### NERDTree NERDTree is a great plugin for beginners and also for people doing competitive programming as it saves a bit of time as well. You can install the NERDTree plugin from the GitHub docs. You can read about the detail of configuring the NERDTree plugin in [this article](https://mr-destructive.github.io/techstructive-blog/vim/2021/08/31/Vim-NERDTree.html). This plugin allows you to have a graphical representation of the files and folder structures in the current directory. You can extend its capabilities to auto-refresh and auto open/close using some custom configuration as provided in the above-mentioned article. You basically open the NERDTree with the command `:NERDTree` after installing and configuring. You can make key-bindings and shortcuts to use as you like and prefer. ### Autocompletion and LSP Autocompletion is quite important when it comes to competitive programming as you have to save time on the **stuff you know** and let the application process it for you. We do have some plugins for auto-completion and also LSPs for this specific task. LSP is quite great and provides much more accurate predictions when it comes to autocompletion. You can check out the LSP [unofficial plugin for vim](https://github.com/prabirshrestha/vim-lsp) for more information. For normal plugins like [AuotoComplPop](https://github.com/vim-scripts/AutoComplPop) are also decent and provide a great beginner experience. Not many plugins would be required for competitive programming just some autocompletion and a nice interface with a file system will work for the basic set-up but that is not the end, you can extend it as per your needs and skills. Vim is quite epic in terms of the customization it can offer.  Similarly, for C++, I have demonstrated the use of key mappings and also showed the usage of the terminal inside of Vim, this is quite a flawless experience to integrate and use the terminal as per choice and preference. ## Conclusion So from this article, we were able to set up Vim for competitive programming by adding some custom key mappings and plugins for different languages. This was not a complete guide, I could not include every language's configuration here, surely you can ask me anytime in the comments, GitHub, Twitter or just mail me I can try to configure a programming language on Vim. Thanks for reading. Happy Viming and Happy Coding :) https://www.meetgor.com/posts/find-filter-github-issues Introduction Are you stuck in finding an open-source project to contribute to? We will see how you can pick up an issue on GitHub appropriate as per your prefer Thu, 09 Sep 2021 00:00:00 UTC ## Introduction Are you stuck in finding an open-source project to contribute to? We will see how you can pick up an issue on GitHub appropriate as per your preferences of languages, labels, complexity, and thus you can find a Community or a project to work and continue with further contributions. This process might not be as efficient but is quite helpful for beginners or people getting started to contributing to Open Source. ## Understand the search bar I assume you have your GitHub account already created. If not go ahead at [Github](https://github.com/join) and create one. On the Home page, you can easily navigate to the `Issues` tab and you will see something like this:  Now, you won't find any issues if you haven't created any. But if you look at the search bar, you will find the reason why it is empty or why there are only the issues that you have created. You will see that in the search bar there is a filter called `author:Username`, which filters the issues which are created by you. You definitely don't want this as you want to search and find other issues by other people/communities. So, simply remove the text `author:Username` from the search bar. Keep rest as it is for now. Now if you press enter after removing the author filter, you will see all the issues on GitHub.  There will be a ton of them, very random in terms of programming languages, frameworks, projects, difficulty, type, etc. they are basically the issues created recently on GitHub. In the next section, we will see how to filter those issues as per the programming languages/tools to which you might like to contribute to. ## Add languages We can add filters to the issues as `language:name`, this will filter all the Issues which have the languages in their codebase. For Example:  Here, I have filtered the issues which have language as `python`, you can use any language/tool you might want and would love to find some interesting projects to contribute and learn from. If you want to search by multiple programming languages you can separate the names of those programming languages by a comma `,`. You can also separate programming languages with space and enclosing all of them under double quotes `""`. For Example: Let's search for issues with C, C++, and Java as their programming languages, we can use `language:c,cpp,java` or `language:"c cpp java"` The above filter will give out all the issues which are created from programming languages either C/C++/Java. You can find more filter options on the [GitHub docs](https://docs.github.com/en/github/searching-for-information-on-github/searching-on-github/searching-issues-and-pull-requests). ## Add labels You can find issues as per labels marked on them, many issues have a label marked on them to improve their visibility and meta-information about the issue. We have some labels which GitHub has created already for common scenarios in projects. 1. `bug` 2. `documentation` 3. `duplicate` 4. `enhancement` 5. `good first Issue` 6. `help wanted` 7. `invalid` 8. `question` 9. `wontfix` We can even create our own labels by providing the label name and a description. To search for labels, you can use `label:name of the label`. You can any of the above 9 label tags or any other tag name that you think is popular other than those 9. You would have to use double quotes (`""`) to add certain labels with multiple words like `good first issue` or `help wanted`. For example: If you search for `label:"good first issue"`, you will get all of the issues(newest first) which have a label `good first issues` tagged on them. Similarly, for multiple issues, you can add comma-separated labels as well. Just like `label:bug,"good first issue"` will search for either `bug`, `good first issue` or both.  ## More Sorting Options In the rightmost part of the search bar, in the Sort button, you can click on there and find a couple of options like: `newest`, `oldest`, `least commented`, `recently updated`, and so on. If you click on any of them you will see the changes reflected on the list of issues as well as the search bar.  **After this the stage is yours, you can look at any issue and Understand its objective, then ask yourself can you solve this issue? If yes then read the contribution guidelines, and the rest is sheer skills like git, programming, documentation, etc.** ## Conclusion Now you can go ahead and start applying the filters on issues and make some contributions to Open-Source on GitHub. We covered some methods and tricks to find and filter out the issues on GitHub based on the programming languages/tools and the labels attached to them. This technique can be good for beginners as well as people who want to find quick issues to solve. Feel free to explore and try out different filters and find the issue you are confident to work on. Good Luck! Happy Coding :) https://www.meetgor.com/posts/why-use-vim Introduction So, Why would one use Vim? As Vim being the most complicated Text editor when it comes to **closing it**!! Also, people think it's not worth the ti Sun, 05 Sep 2021 00:00:00 UTC ## Introduction So, Why would one use Vim? As Vim being the most complicated Text editor when it comes to **closing it**!! Also, people think it's not worth the time or it's just not their type of Text Editor. Well, that's true, I am not forcing anyone to use Vim, just to tell the reasons why people use Vim. Again, it's not to show off to anyone that I use Vim, just to appreciate the power of Vim and its community, it's just amazing!! Vim is not a text editor just for anyone, it can be daunting initially but the more you use it, the more you love it. There is nothing called a time-wastage in learning Vim, if you think learning something is waste of time, just don't learn it! To have understood and got good at something you need to dedicate time and effort to it, there's no shortcut. When it comes to shortcuts, Vim is the fairyland of it, just use one or create your own. That's how cool Vim can be when used properly. Some of the most important and crucial reasons why people use Vim are: ## Vim is Speed! Vim can be opened from the terminal, and that itself can speak that it is lightweight and uses lower system resources and hence causing minimal lag. This might not be an issue for high-end computers but when it comes to low-end machines, Vim behaves as a life-saver. Using certain commands feels so wicked fast and intuitive. Certain commands such as `t` or `f`, `u`, and many others are really powerful when it comes to editing in Vim. When you don't have to think about what keys you should type, you are really getting faster in Vim.  The above screencast is from my article [Vim: Set up for Python](https://mr-destructive.github.io/techstructive-blog/python/vim/2021/06/06/Vim-for-Python.html), which uses some key mappings to run python code directly from Vim itself by executing the shell commands from Vim's Command mode. A simple map can save a lot of time. ``` nnoremap py :!python % ``` ## Number of Commands/shortcuts When it comes to using key-bindings and shortcuts, there is no shortage of them. You can spend days, weeks, months, years but still, there will be some or the other thing to learn in Vim. That can be very intimidating to some people but that's how life is. **You cannot get perfect at anything but surely better is the word to focus on.** There is a good little cheatsheet at [devhints](https://devhints.io/vim) for beginners to quickly get started. Also, you can follow with the Youtuber- [ThePrimeagen](https://www.youtube.com/channel/UC8ENHE5xdFSwx71u3fDH5Xw), who focuses on switching and using Vim. Vim is has a robust set of command and key-bindings by far compared to other tex-editors. Here are some of my daily used [editing commands](https://mr-destructive.github.io/techstructive-blog/vim/2021/07/18/Vim-Enhancing-Editing-speed.html) and [navigation commands](https://mr-destructive.github.io/techstructive-blog/vim/2021/06/26/Vim-Enhancing-Movement-speed.html). ## Power to use it as an IDE Vim provides some awesome features such as [terminal integration](https://mr-destructive.github.io/techstructive-blog/vim/2021/06/29/Vim-Terminal.html), [Key-mappings](https://mr-destructive.github.io/techstructive-blog/vim/2021/06/14/Vim-Keymapping.html), [Window Splits](https://mr-destructive.github.io/techstructive-blog/vim/2021/08/06/Vim-Window-Splits.html), etc to take text-editing and developer environment to a next-level. You can literally use Vim as an IDE for many of the programming languages like C/C++, Python, Java, Javascript, GO, etc. Though it can't replace entire IDE-like features, it is indeed powerful enough to add features in making a custom IDE experience. It might require some time and research to configure it to use it as a full-blown IDE, but it might be a very useful and self-motivating experience. There are definitely many articles, tutorials out to guide you in configuring the editor to an IDE. People might not use Vim for the features it has but also for the lightweights as compared to other IDEs such as Visual Studio, Android Studio, Eclipse, IntelliJ IDEA, etc.  From, the above screenshot, we can see that I have configured Vim almost like VSCode, Ok! Just the look of it is similar but ya we have the file manager as [NERDTree](https://mr-destructive.github.io/techstructive-blog/vim/2021/08/31/Vim-NERDTree.html), integrated Terminal, and the buffer opened as the editor. You can even add more functionality like autocomplete, keymaps to compile/run/build the code. There is are infinite possibilities when it comes to customization in Vim and to make it a full-blown IDE. OH! And by the way my Vim look changes from time to time XD ## Available almost everywhere As said, Vim can be opened via the terminal, and hence it is quite viable when it comes to accessibility. It is pre-installed in almost every Linux-distributions, macOS, FreeBSD, etc. Hence it is quite reliable when it comes to compatibility and cross-platform support. You can start editing in any environment in Vim, so it becomes a good reason to learn to use Vim at least basic editing and opening, closing, etc. Vim being minimal and lightweight, it just becomes easy to work with servers. Production-environment, etc. It also isn't that complicated once you make your head around it. Being available in almost every Operating System has some value to it like it is the most dependable editor out there. Even after 30 years, yes Vim was released in 1991, it is still maintained by its producer [Brain Moolenaar](https://en.wikipedia.org/wiki/Bram_Moolenaar). That is mind-boggling, Vim is truly a vintage text editor silently but widely dominated in terms of its reach in Computing. ## The Level of Customization/Configuration It's kind of a double-sided sword for some people as it can be daunting to customize from scratch and also, it could be a rewarding and pleasurable experience of having the personalized experience. With the number of plugins, custom vimrc and color schemes, etc Vim is very prone to personalization. It is **Open Source**, so we also have it's distributions such as [SpaceVim](https://github.com/SpaceVim/SpaceVim), [SPF-13](https://github.com/spf13/spf13-vim), etc. Some more distribution can be found [here](https://dev.to/ajeebkp23/some-popular-vim-distributions-a68). Some Distribution is pref-configured with some basic stuff and is ready to use, so they are quite beginner-friendly and introduce them to Vim in a better way. NeoVim which is an evolution of Vim written in Lua is much more powerful and provides even more customization keeping the basic key-bindings in mind. Editing Vimrc is quite an amazing and intuitive experience as it allows to add functionality or enhance the existing features in Vim. ## Conclusion > Text Editors don't matter, what matters are the problem-solving skills Vim is a simple and minimal text editor, everyone can use it but not everyone finds the need to use it, that's fine. It might be ideal for me to use Vim than other editors but it might not be for you. But there is nothing to lose in knowing some basics of Vim. It might be useful somewhere, you never know! And if your favorite editor is not available, Vim might prove to be a great quick solution. Thanks for reading. Happing Coding :) https://www.meetgor.com/posts/vim-nerdtree Introduction is a great plugin in Vim for managing and navigating Files. Some might prefer fzf, telescope, and other plugins for navigation, NERDTree is not a b Tue, 31 Aug 2021 00:00:00 UTC ## Introduction [NERDTree](https://github.com/preservim/nerdtree) is a great plugin in Vim for managing and navigating Files. Some might prefer fzf, telescope, and other plugins for navigation, NERDTree is not a bad option to begin within Vim. NERDTree allows you to even create/delete/move files and folders flawlessly without much effort, so it becomes a much more viable beginner's plugin. ## Installing NERDTree Plugin So, let's start with the Installation of the NERDTree Plugin, it's quite straightforward and simple. You should have a Plugin-Manager for installing this plugin. It's not mandatory to have one but it becomes much easier to work with. You can choose any between `Vundle`, `Vim-Plug`, `Pathogen` to name a few. It does not really matter with what you use just stick to one but if you are stuck somewhere just switch and it's not a big trouble to use other Plugin Managers as they are quite similar to each other. #### Vundle To install a plugin using Vundle, you need to configure Vundle first if you have not already done it. You can find the installation docs [here](https://github.com/VundleVim/Vundle.vim). After Vundle has been configured in your vimrc you can simply add `Plugin 'preservim/nerdtree'` between the call begin and end of Vundle, like : ```vim call vundle#begin() Plugin 'preservim/nerdtree' call vundle#end() ``` All of your other Plugins will go in between those two statements, i.e. `call vundle#begin()` and `call vundle#end()`. After saving and sourcing the vimrc file, you need to install the plugin using the command `:PluginInstall`, and there you are all Done! #### Vim-Plug To install a plugin using the Vim-Plug manager, you need to configure Vim-Plug if you have not already configured it in your vimrc. You can find the installation docs at the GitHub README of [Vim-Plug](https://github.com/junegunn/vim-plug). After Vim-Plug has been configured in your vimrc you can simply add `Plug 'preservim/nerdtree'` between the call plug begin and end statements. Just like: ```vim call plug#begin() Plug 'preservim/nerdtree' call plug#end() ``` All of your other Plugins will go in between those two statements, i.e. `call plug#begin()` and `call plug#end()`. After saving and sourcing your vimrc file, you need to now install those plugins using the command `:PlugInstall`, and that is it! #### Pathogen To install any plugin using Pathogen plugin manager, you need to configure Pathogen in your vimrc if you have not done it already. You can find the installation docs on [Pathogen.vim](https://github.com/tpope/vim-pathogen). After Pathogen has been configured in your vimrc, you can clone the git repository of that plugin into your local machine and then activate it using Pathogen. ``` git clone https://github.com/preservim/nerdtree.git ~/.vim/bundle/nerdtree ``` After cloning the repository, you can add this to your vimrc where you have configured it. It's a kind of DIY manager in terms of managing the folders of the plugin. ```vim call plug#begin() call pathogen#infect() syntax on filetype plugin indent on ``` After this, you need to run this command to get docs and help with the plugins, `:help tags ~/.vim/bundle/nerdtree/doc/` or `:help tags` And there you are done with the plugin installed. There are other Plugin managers as well, but these three are the most widely supported ones and they work out of the box, surely explore for yourself and find the perfect one for you. ## Activating and Using NERDTree Now, we actually need to use NERDTree, for that we can type in `:NERDTree` in any folder in our local machine, and there should be a window open a vertical split to the left, just like this:  After this, you can use <C-w> that is **CTRL+W twice** twice to switch back and forth between windows. You can also use **CTRL+W and HJKL** to move in directions in the windows. For further guides and key-bindings in Window-Splits, you can read my article [here](https://mr-destructive.github.io/techstructive-blog/vim/2021/08/06/Vim-Window-Splits.html). Now, you can navigate to the file/folders using HJKL or arrows keys(not preferred). You can even use the numbers before the HJKL to jump and hop around the large codebases, this integrates really well with base Vim key-bindings. You can quiet the NERDTree window by just pressing `q` or `:q`, definitely the former is efficient. You can open/collapse the folders also using the enter key to open the file in the current buffer. But hey that's quite limited, what have you ever seen! #### Open File in Splits You can open a file in a horizontal split using the key `i` on the file. You can open a file in Vertical split using the `s` key keeping the current highlight in NERDTree on the file which you would like to open. This can be really a great feature to have while opening multiple files and file structures. #### Managing Files/Folders using NERDTree You can create files using the NERDTree window by pressing m inside the particular folder where you want to. If you want to create a file in the root folder, you can go to the topmost file location and press `m` inside the NERDTree window. If you press `m`, you will be able to see different kinds of options namely: 1. Add a child node.(`a`) We can create a file or a folder using the key `a` or simply `Enter` to create the file in the current highlighted location. 2. Move the Current Node. (`m`) We can create a file or a folder using the key `a` or simply `Enter` to create the file in the current highlighted location. 3. Delete the current Node. (`d`) We can move the currently highlighted file/folder to any other directory using the file manager itself. 4. Open the current Node in the system Text-Editor.(`o`) We can delete the file/folder which is currently selected on the NERDTree menu. 5. Copy the current Node. (`c`) We can open the file in the system-default text-editor using the key `o`. 6. Copy the Path to the clipboard.(`p`) We can copy the current file/folder or a node using the command `c`. 7. List the Current Node. (`l`) We can list the file/folder i.e to display its properties the read/write/execute permissions, date modified and created, etc. 8. Run system Command in this folder. (`s`) We can run system commands or shell/terminal commands using the key `s`, For windows, we open the COMMAND PROMPT, and in Linux and macOS, it is terminal. You can quit that window by pressing `Esc`. Here is some of the Screencast of me demonstrating the NERDTree plugin features and the edit options.  This is just self-explanatory and beginner-friendly sets of commands, it becomes really easy to do this with some custom macros as we'll talk about in the next sections. ## Macros to open file tree You can make a key-binding to open the NERDTree, ```vim nnoremap <C-n> :NERDTree<CR> ``` You can map anything instead of `<C-n>`, most people use `<leader>` but it's easy to use `CTRL+N` for me, it's just personal preference. If you do not like to open NERDTree again and again, you can keep it open whatsoever using the custom key-binding in your vimrc. ```vim autocmd VimEnter * NERDTree ``` This will open the NERDTree automatically for you when you open Vim, Ya I get it, it's not needed every time but most of the time a developer is switching between the files. ## Enabling Autoreload We can auto-reload the NERDTree window when there is a change in the file structure, i.e. a file/folder is deleted/created/moved/etc. We again need to set this in our vimrc: ```vim autocmd BufEnter NERD_tree_* | execute 'normal R' au CursorHold * if exists("t:NerdTreeBufName") | call <SNR>15_refreshRoot() | endif ``` This will reload the NERDTree when the cursor is in the NERDTree's window. This could be really time-saving and a nice quick configuration to enhance the efficiency of your Text-editing. ## Enabling Autorefresh for change in the current directory We can also reload the NERDTree when we change the directory. The above-mentioned command is not sufficient to do that, we have to add another set of configurations. ```vim augroup DIRCHANGE au! autocmd DirChanged global :NERDTreeCWD augroup END ``` By adding this to your vimrc, you will refresh the NERDTree every time you enter or change the current directory. This is also a great addition to have to save time by reloading the Window for every change in the path, if you are looking for something among a huge code-base, this works a charm. ## Auto close You need to close the NERDTree manually each time you want to exit out of it, but this can also be automated just for the sake of simplicity and effectiveness in **QUITTING VIM**. ```vim autocmd bufenter * if (winnr("$") == 1 && exists("b:NERDTree") && b:NERDTree.isTabTree()) | q | endif ``` This will close the NERDTree window if it is the only open window. That can be frustrating at moments but the majority of the time this is a great addon indeed. ## Packing it together So, we have learned the basics of using and modifying NERDTree according to our needs, to put it together, you can use this snippet directly into your vimrc and enjoy the flawless experience. ```vim " Open nerdtree window on opening Vim autocmd VimEnter * NERDTree " Refresh the current folder if any changes autocmd BufEnter NERD_tree_* | execute 'normal R' au CursorHold * if exists("t:NerdTreeBufName") | call <SNR>15_refreshRoot() | endif "Reload the window if directory is changed augroup DIRCHANGE au! autocmd DirChanged global :NERDTreeCWD augroup END "Close nerdtree automatically if it is theonly window open autocmd bufenter * if (winnr("$") == 1 && exists("b:NERDTree") && b:NERDTree.isTabTree()) | q | endif ``` ## Conclusion: So, We were able to make Vim a better place to work with. Making it easier to navigate along with files and folders. Configuring the NERDTree Plugin, customizing the look and functionality of Vim as per the needs. NERDTree is a great plugin no matter how you use it. It makes Vim more viable as a text editor for daily use and that also in an efficient and clean way. Surely there might be other plugins that are super powerful and blazing fast, NERDTree provides a good UI as well by providing a graphical representation of the File structure that enhances its usage. That is what Vim is about, learning every day some things to change the way to edit. Thank you for reading. Happy Viming and Coding :) ### References: - [NERDTree - docs](https://github.com/preservim/nerdtree) - [Refresh NERDTree](https://stackoverflow.com/questions/8793489/nerdtree-reload-new-files/8794468) - [Reload NERDTree on Directory change](https://vi.stackexchange.com/questions/31050/how-can-i-make-nerdtree-update-root-to-the-current-directory-when-i-change-direc) - [Open NERDTree in Vim by default](https://stackoverflow.com/questions/1447334/how-to-add-nerdtree-to-your-vimrc) - [Close NERDTree automatically](https://stackoverflow.com/questions/2066590/automatically-quit-vim-if-nerdtree-is-last-and-only-buffer) https://www.meetgor.com/posts/devquotes-platform Introduction No one can understand the joy in finishing a personal project, except the dreamer It was a while,since I have posted an article here, as I was busy Fri, 27 Aug 2021 00:00:00 UTC ## Introduction > No one can understand the joy in finishing a personal project, except the dreamer It was a while,since I have posted an article here, as I was busy on a project or a hackathon. Hello, world! I am Meet a student and a self-taught web developer. I like to make and break stuff, especially when it comes to programming and Linux. I like shell scripting and learning different languages at once, love to learn about Vim and Linux everyday. Every time I start a project something else comes and distracts me let that be any other programming language or technology. That leads to creating new projects and leaving the one behind unfinished, I know most of the developers face this. But this time, thanks to Auth0 X Hashnode Hackathon, I was able to create an almost finished project within almost 10 days. Having a deadline and competition creates a mindset to finish a project on time, that's my first takeaway from this Hackathon. OH! this is my first Hackathon by the way, and it has been amazing so far. ** Applying a framework to do something you desire and then everything working smoothly (after fixing 100s of bugs) is such a great feeling that no one can understand except for the person who just dreamt of it. ** I'll like to share my project which is a web application for the Auth0 x Hashnode Hackathon. Here it goes. ## What is Dev Quotes? Dev quotes is a web app designed for publishing and viewing quotes related to programming, developer mindset, and all the technicalities involved in a developer's life. It's basically a medium to express the life of developers and get inspired by others. Here it is [devquotes](https://devquotess.herokuapp.com/) #### Dark Mode:  #### Light Mode:  ## Why Dev Quotes? > Developers are not the people who only understand how to write code but they're also the people who can make the code understandable As a developer, there are often times where you have no motivation left inside, but you never know you might be just a few lines of code away from making another project or fixing a bug. For that, we require some inspiration or a push to break the barrier of. I am not saying it's just for developers, it's designed for developers but everyone is open to understanding the developers' lives and their struggles. I also felt the need to give back some love-crafted web app to the ever-wonderful and supportive dev community. It's a small application but still, I would like to give in something instead of nothing at all. Start small grow big, hopefully :) ## Features Some of the main features of the web application are as follows: - **Write�dit\Delete Quotes if logged in.** - **Like / Unlike a Quote.** - **See all of your quotes.** - **Randomized Quotes on Homepage.** - **Dark/Light theme based on Browser's Preference and local storage.** - **The app is mobile responsive as well, though the navbar is a bit wonky with the light/dark mode switch toggle, which can be taken care of soon.**  ## In the making > Have the curiosity to learn, rest is automated The project was made by using various inspirational articles and videos about making a web application. But the idea stuck in my mind when I was thinking about the people who don't get inspired as a developer. Like there is no way you can remain sad about being a developer and keep on dealing with imposter syndrome. Every developer has a perspective of programming but there is an infinite number of opportunities if you are curious enough. Just started making the project and got so much into it that I was literally dreaming about it like I saw parts of the webpage. In my dream and I am making it that was genuinely a thing that powered me to complete it. The project roughly started on 19th August and almost ended on 26th August, like the actual webpage and its core functionalities. Though on 27th were some styling and extra additions such as the About section and Footer. That was like the most productive week I ever had in my programming journey. That was fun as heck. ## Tech Stack The Tech Stack involved with this app is : - `Django` - `PostgreSQL` - `HTML/CSS/JS` - ` Bootstrap` I have not used any front-end end frameworks just because I never found the need to learn them. I had experience with Django for just 2 months and I am surprised I was able to make it. As obvious I have used Auth0 for authentication in my web application. ### Auth0 integration for Authentication I must tell you using Auth0 was just flawless addition to my app as I have to do almost nothing, just drop some credentials of the Auth0 application with my Django project using a [well-documented guide](https://auth0.com/docs/quickstarts) for every type of framework. Simply straight-forward was the name for integrating authentication in my app. #### How I used Auth0 with Django I've used Template tags such as if blocks to verify if the user is authenticated or not. ```html <div class="login-box auth0-box before"> {{ "{% if user.is_authenticated "}} %} <a class="btn btn-primary btn-sm tn-logout " href="#">Log Out</a> {{ "{% else "}} %} <a class="btn btn-primary btn-sm tn-login " href="#">Log In</a> {{ "{% endif "}} %} </div> ``` This was just readily available on their documentation though there were some adjustments as per the project requirements in this code to fit in the place. I must say, integrating Auth0 is even easier than using Django User Model in some sense as most of the stuff is handled by the Auth0, on our side, we simply have to create the Auth0 specific app with the credentials from the dashboard rest just works flawlessly till now. How sweet and ### Specifications I won't go in-depth about the technicalities of the project but would like to address certain things. Firstly I have mostly used Class-based views for the major part, certain areas are still function-based just for the simplicity of the application and a few of them are handled and documented by Auth0 so just preferred that. Another thing is about Models, I just have a simple single model called `Quote` which has an Author as a Foreign Key from the Django User Model. I would have also created multiple emojis for the like system but I was too excited and in a rush to see the actual app, So just kept it simple. XD  The rest of the stuff like `URLs`, `Templates`, and `static files` are handled in a neatly organized way as depicted in the below diagram.  As it might not be clear from that, there are 3 apps -> `auth0login`, `quotes`, and `user`, here `quotes` is kind of the most important app as it has the models, forms, URLs, and the views linked to them here. ### Hosting Hosting as you can guess, it's on [Heroku](https://www.heroku.com/) , quite beginner-friendly and feature-rich. I also have a free addon for PostgreSQL Database here. It's limited to 10K rows but that's sufficient for a starter app like this in my opinion. Also, it has 500 free hours of dyno, which is also sufficient for a small low-traffic app like this.  ## Bugs Encountered > Love Bugs, they'll give you experience This is the most exciting and important part if you consider the hackathon because this determines the level of experience that a developer shoes in achieving certain things or features if you will. Faced some typical Django errors as usual but also some unexpected things like Dark mode and light mode clashing together due to poorly written media queries in CSS. As usual, the start is often hard, there is no motivation whatsoever in using the admin section to test the database queries and gibberish basic HTML page. In that process, I faced some primary key access issues and was able to understand the concept more clearly by fixing it. Another instance was with handling post requests in Django which I've failed to do before. I used forms and a hybrid of CSS and bootstrap to style those forms which just works brilliantly. That took some time to figure out the exact working but after a while, it was working charms. ## Future Updates As said, I would like to add more like buttons expressing different emotions. Some other features to add are: - To add more emojis like claps, cheers, and others. - To add a profile page in the app that would display all the quotes of the particular author and the details related to him/her. - Adding some tags to filter out particular types of quotes. - Improve UI-UX a bit more to make it professional and pleasant. ## Source Code >Talk is cheap, show me the code - Linus Torvalds The source code is available at GitHub on this [Link](https://github.com/Mr-Destructive/devquotes). It's freely open for any contribution after the hackathon(mid-September). Some of the files such as the environment variables, virtual environments, cached data are not uploaded for security and obvious reasons. Enough of technical talks, let's see the DEMO,  Silent claps....... It's not a great UI-UX but works and is not too shabby in my opinion considering it only has base HTML and CSS with a little Bootstrap and Javascript. But ya, a fully functional Backend that's what I was looking for a full-stack app with some decent features. Hope it helps some developers stay motivated and hack into some hackathons like this. ### References used while creating the app: - [Codemy -John Elder Django tutorial]( https://www.youtube.com/watch?v=B40bteAMM_M&list=PLCC34OHNcOtr025c1kHSPrnP18YPB-NFi) - [Django - Documentation for Forms](https://docs.djangoproject.com/en/3.2/topics/forms/) - [Django template randomizer shuffle](https://stackoverflow.com/questions/28837511/django-template-how-to-randomize-order-when-populating-page-with-objects) - [Auth0 app Django integration](https://www.youtube.com/watch?v=kzN_VCFG9NM) ## Closing Words > Why developers find solutions to bugs at the stroke of sleeping, that's multithreading in our brains Hope you liked the project and hopefully will inspire developers to stay motivated and can focus on their goals more than dealing with imposter syndrome and whatnot. Thank you for reading and using the app, for any feedbacks, Twitter handles, comment section, GitHub issues, LinkedIn messages are all freely open. Thanks. Happy Coding :) https://www.meetgor.com/posts/django-setup-script Introduction Once you learn something very neatly and understand it very clearly, it feels like you are wasting time doing things over and over again. Same thin Sun, 15 Aug 2021 00:00:00 UTC ## Introduction Once you learn something very neatly and understand it very clearly, it feels like you are wasting time doing things over and over again. Same things in setting up a Django project, you feel like you could save those typing minutes by making a script. In this article, we will make a script a batch script for windows and a BASH script for Linux/macOS. We will make a virtual environment using python and then install the libraries and dependencies like in this case we will install Django. You can also tinker with these scripts and install other dependencies if you want like Django rest framework, crispy forms, markdown, Redis, etc. We will also make a Django project using the positional parameter passed before running the script from the command line. ### Python development environment This article assumes you have a python environment setup. If you don't you must install Python from the [official website](https://www.python.org/downloads/) as per your operating system. Also, you should have pip installed and configured correctly. You can install pip by following the [official documentation](https://pip.pypa.io/en/stable/) for the specific operating systems. ## Steps in Django Project Setup So, If you are already familiar with the Django project setup, you can directly use the scripts provided in the next few sections. You can also visit [this GitHub repository](https://github.com/Mr-Destructive/django-quick-setup-script) if you have any issues and errors. If you are new to django, let me first explain the process of django project setup. - ### Initialize a VirtualEnvironment (Recommended but not necessary) Virtual Environment in Python is a great way of localizing the dependencies and frameworks only in the particular folder, it allows the developer to separate things out and keep them distinct, Additionally, when deploying or sharing the repository, the other developers can install the dependencies in the requirement.txt file in their local environment flawlessly. So, it is always recommended to use python virtualenv when working with python frameworks or libraries. We can set it up by simple pip install and then giving it a name. ``` pip install virtualenv ``` This will install the package/tool using pip. After that has been properly installed, we can now give it an appropriate name ``` virtualenv mytest ``` The `virtualenv` is the command and `mytest` can be any name, generally `env` or `venv` is preferred but it should be understandable to the user. You will now be able to see the folder inside of your current directory named as `mytest` or the name you've given to it. **Windows** Now if you are on windows, you can activate the virtual environment by using the command : ``` mytest\Scripts�ctivate ``` here mytest is the name of your virtual env it can be anything as per your choice. This will now activate the virtualenv which will be shown by `(mytest)` before the command prompt. **Linux/macOS** For Linux or macOS, you can use the command: ``` source mytest/Scripts/activate.sh ``` In the above command, `mytest` can be anything that you have used while creating the virtualenv. This should activate the vrtualenv and will be indicated by `(mytest)` before the prompt in the terminal.  From the above image, we can see that we created and activated an virtualenv in python in a folder. So, this is the walkthrough for setting up the virtualenv for a Django project, now we will move to install Django in this environment. - ### Installing Django using pip This is quite straightforward. You can use `pip install django` or `pip3 install django` or the normal way you install a library from pip. - ### Creating a Django project To create a django project, we use the django-admin command like: ``` django-admin startproject mywebsite ``` This will create a folder called `mywebsite` or your project name anything you like. Inside the `mywebsite` folder, you will have 2 things: `manage.py` file, and `mywebsite` folder . Yes there will be another `mywebsite` folder inside your project which will have the settings, URLs and other global(project-level) configuration files. The `manage.py` file is the most important file here. You should never touch/edit this file. We use this file to carry out all the operations from running the server to managing the database.  ### Setup script for Windows The below is a batch file for Windows Operating System. Save the file in a `.bat` extension. <iframe src="https://carbon.now.sh/embed?bg=rgba%28171%2C+184%2C+195%2C+1%29&t=cobalt&wt=none&l=application%2Fx-sh&ds=true&dsyoff=20px&dsblur=68px&wc=true&wa=true&pv=56px&ph=56px&ln=false&fl=1&fm=Hack&fs=14px&lh=133%25&si=false&es=2x&wm=false&code=mkdir%2520%25251%2520%250Acd%2520%25251%250Apip%2520install%2520virtualenv%250Avirtualenv%2520env%250Acall%2520env%255CScripts%255Cactivate%250A%250Apip%2520install%2520django%250Adjango-admin%2520startproject%2520%25251%2520.%250Acls%250A" style="width: 803px; height: 366px; border:0; transform: scale(1); overflow:hidden;" sandbox="allow-scripts allow-same-origin"> </iframe> Make sure the file is saved in a `.bat` file and be in the folder where you would like to create the Django project. After being in the appropriate location, enter the following command: ``` commands.bat myproj ``` Here I assume that you have named the script file as `commands.bat`, you can name it anything you like, but I like to keep this a convention. After this you don't need to do anything, everything will be handled by the script. You can run the server using ``` python manage.py runserver ``` This will have the base django project set up on your system. The below is the live demonstration of the script, I have deliberately removed the `cls` command to show the process. It won't break the script if you add this to it.  ### Setup script for Linux/macOS Copy the code from the below script and save it in a file preferably called `commands.sh`, you can name it anything you want but keep the `.sh` extension after it to identify it as a shell-script. After that file is saved locally, you can run the script by passing the positional parameter as the name of the Django project. The command will be like: ``` bash commands.sh myproj ``` <iframe src="https://carbon.now.sh/embed?bg=rgba%28171%2C+184%2C+195%2C+1%29&t=blackboard&wt=none&l=application%2Fx-sh&ds=true&dsyoff=20px&dsblur=68px&wc=true&wa=true&pv=56px&ph=56px&ln=false&fl=1&fm=Hack&fs=14px&lh=133%25&si=false&es=2x&wm=false&code=%2523%21%252Fusr%252Fbin%252Fenv%2520bash%250A%250Amkdir%2520%25241%250Acd%2520%25241%250Apip%2520install%2520virtualenv%250Avirtualenv%2520env%250Asource%2520env%255Cbin%255Cactivate%250A%250Apip%2520install%2520django%250Adjango-admin%2520startproject%2520%25241%2520.%250Aclear" style="width: 803px; height: 384px; border:0; transform: scale(1); overflow:hidden;" sandbox="allow-scripts allow-same-origin"> </iframe> From the output of the script, you will have created a Django project inside a virtual environment. We can manually activate the virtual environment. You can experiment it within your system as it can be a lot more customizable. This is just bare bone script to start a Django project but you can add your own things into it. ## Conclusion Thus, from this little article, you can get a bit lazier in initializing a bare-bone Django project. We were able to understand the structure of the Django project and how to set up a virtual environment powered by Python. After understanding those concepts we then moved on to making a script namely a batch file and a shell script to automate the initialization of the Django project. I hope it might have helped in some or another way, Thanks for reading till here. Happy Coding :) https://www.meetgor.com/posts/vim-buffers Introduction I have talked about Vim , in the previous articles, and now I am quite keen on explaining the finest unit of file that you can open using Vim and t Wed, 11 Aug 2021 00:00:00 UTC ## Introduction I have talked about Vim [tabs](https://mr-destructive.github.io/techstructive-blog/vim/2021/08/03/Vim-Tabs.html), [window splits](https://mr-destructive.github.io/techstructive-blog/vim/2021/08/06/Vim-Window-Splits.html) in the previous articles, and now I am quite keen on explaining the finest unit of file that you can open using Vim and that is a buffer. Tabs are a collection of windows, Windows are the viewport on buffers (collection to view the buffers), and Buffers are the memory that holds text in the file. So let us explore buffer in detail. This article won't cover each and every tiny detail about buffers but surely enough to make you understand what a buffer is and how to manage those. ## What is a Vim buffer Vim buffers are the chunks of memory that hold the text in a file. Vim buffers are used from opening a file to using, installing, upgrading the Plugins, using file managers, and everything you can edit and work within Vim. So, why bother using Vim buffer, you'll ask. Well, you will need it to make some complex tasks possible in Vim, such as configuring it as your IDE or having much more control over what and how you open and close files in Vim. You might not need it in the broader sense but having a grasp of what you are using and understanding the process under the hood is vital whatsoever. ## Creating a buffer No need to create a buffer, if you are editing a file, that's already a buffer :) If you open Vim, a fresh open, then you already have that current buffer as an empty buffer. You can create an empty buffer by entering the command `:enew`. This will actually make a new empty buffer, if you have edited the contents of the previous buffer you need to specify to write/save the buffer in a file. You can see the details of the buffers opened currently by using the command `:ls` or `:buffers`. This will open a list of buffers currently loaded in the Vim instance. You will see the output somewhat like this:  We can see the name of the current buffer which in this case it is `"No name"` this will be replaced with a file name if we had a file open. This won't only print the current buffer but all the loaded buffers in the present Window of Vim. We will explore the details of this command in the next section. ## Navigating through the buffers list  In the first column in the only row, you can see `1` indicating the number of the buffer, secondly, there are indicators `%a` and `#` and others as well. These are used for referencing the buffers using certain attributes those have like : - `%` stands for the buffer in the current window. - `a` stands for the currently active buffer. - `#` stands for the alternate buffer (previously edited). - `h` stands for the hidden buffer ( used for navigating to other files without saving changes ). We also have additional attributes or indicators for the buffers like: - `+` indicating the buffer is modified. - `-` indicating the buffer cannot be modified. - `=` indicating the buffer is read only. To use the attribute `h`, you have to set the option in the current window only, you can use `:set hiddden` to hide the files i.e to edit another file without saving changes to the current buffer(with the file loaded). If you want a permanent option of hidden files you can add `set hidden` in your `vimrc` file.  If you have enabled a hidden file in the window and you modify the file, you can see the `+` symbol in the `:ls` command output before the file, indicating that the file has been modified. If you try to quit the window, it will prompt you to save those unsaved modifications. This is the power of Vim buffers, you can get really fast in terms of editing multiple files and saving all of them at once. This feature can allow you to work more effectively and flawlessly with Window Splits and Tabs as they can contain multiple buffers within them. Now we will see how to move between those buffers. We can pretty effectively use the numbers in the buffer list to directly jump to that buffer. We can use `:b n` where n is any number of the buffer in the list of buffers to move through the buffers. Optionally we can use `:buffer n` but that is quite a long command. We can move to the previous buffer as `:bp` to move to the previous buffer in the list. Similarly, we can use `:bn` to move to the next buffer in the list, and `:bfirst` and `:blast` to move to the first and the last buffer respectively.  ## Moving through the buffers We can move through our opened buffers using `Ctrl + o` to move backward a buffer and `Ctrl + i` to move forward a buffer. This can be helpful if you do not want to remember the numbers associated with the buffers.  We can scan through the current buffers opened and jump through them one by one and it becomes quite effective in switching between few files. This could probably be suited for Web development where you can hop between HTML/CSS/JS files or in Android development for java/XML files, and so on you get the idea for using it as per your needs. ## Deleting buffers Now we will see how to delete the loaded buffers, we can simply use `:bd n` where n is the number of the buffer in the list to delete the buffer, optionally we can use `:bd filename` to delete the buffer loaded with that file.  We can also delete a number of buffers at a time using a range of buffers like `:n,mbd` where `n` is the first buffer and `m` is the last buffer. We will delete every buffer between `n` and `m` including `n` and `m` using the above command.  Just a quick fact, if you do not delete buffers, they will stay in memory even if you use `:q` they won't get erased from the buffer list, you are just exiting the buffer view and not removing the buffer from the current memory. This can get really messy if you are opening too many files and keeping them in memory. ## Creating splits in buffer We can even create splits in buffers, basically a window split but we will see this with the view of buffers this time. To create a horizontal split, we can type in `:new filename` to open the file in the split. We can even create a vertical split with `:vnew filename` to open a vertical split with the provided file.  We can see some wired things in the ls menu if we open a new tab in the current instance of Vim. The list command shows all the buffers in the current Tab as well as in the previous tab. We can also open buffers in splits by using numbers or file names. We can use `:sb n` to open the contents of the buffer n(any number of buffer), and with `:sb filename` to open the file in a split this will only be a horizontal split. We can use `:sba` to open all the buffers in the split windows. This can be quite handy to find any file which you cannot remember was edited or wanted at the moment, this opens every buffer loaded in the current instance. We can also use `:sbp` and `:sbn` to open the previous and next buffer respectively in a split. Additionally, we also have `:vertical ba` and `:ba` (`:ball`)to open all the buffers in vertical and horizontal splits respectively.  **Also**, We have `:tab ba` or `:tab ball` to open all the buffers as tabs in the current instance.  You can add more layers of effectiveness or customization using keymaps to split certain buffers and make use of the above set of commands more powerful and make them work according to your needs. ## Conclusion So, from the above sections, we were able to see what happens under the hood when you open, close, or edit a file in Vim. Though it was not an in-depth view of each and every detail of the buffers in Vim. But enough to make you understand what and how buffers work and how to use them to become more effective. Hopefully, you have understood how to perform movements with buffers and create and delete them with tremendous control over what files you could edit. Thank you for reading till here. ### Buffers < Window Splits < Tabs So from these three articles, we were able to understand the terminology of Buffers, Window Splits, and Tabs. These all are the units of the Vim file system/ Way of grouping Files. We can use buffers to load the file from the memory, we can split various buffers into windows which are all contained in a tab. Some might prefer using buffers, splits, or tabs but hey! what are you even talking about? They are all part of tabs so use everything as per need don't force yourself to use only one thing. **Use buffers for opening few files, if you need to open many files regularly use window splits, If that's not enough use tabs to separate from different folders or directories**. Don't stop Viming, Happy Coding :) ### References - [Vim Fandom: Buffers](https://vim.fandom.com/wiki/Buffers) - [Vim Buffers FAQ](https://vim.fandom.com/wiki/Vim_buffer_FAQ) - [Linux Hint - Understanding and using VIM Buffers ](https://linuxhint.com/how_vim_buffers_work/) https://www.meetgor.com/posts/vim-window-splits Introduction Have you ever been stuck in Vim opening multiple files within a single window? Didn't anyone tell you, you can create multiple windows and split th Fri, 06 Aug 2021 00:00:00 UTC ## Introduction Have you ever been stuck in Vim opening multiple files within a single window? Didn't anyone tell you, you can create multiple windows and split them within a single tab. Definitely, the window splits will be in separate buffers. In this way you can create multiple windows inside of a single Tab, what are Tabs? You can learn some basics about it from my previous article about [Tabs in Vim](https://mr-destructive.github.io/techstructive-blog/vim/2021/08/03/Vim-Tabs.html). We can either create Vertical or Horizontal splits within the window making it flexible to work with multiple files in Vim. This article will look into the creation, navigation, closing, and rearrangement of Window Splits. ## Creating a Window Split Creating Window splits is quite straightforward. You should keep in mind the following things though: - You can create a horizontal or a vertical split within a window. - Creating a Split either vertically or horizontally can shorten the current window's size, making it equally spaced. Let's take a look at creating the vertical and horizontal splits one by one: ### Vertical Splits Vertical Split as the name suggests, it will split the current window into **two halves vertically** or a **standing split between two windows**. The below image clearly shows a vertical split between two windows. Here we are splitting a single window into two windows. We can also think it of in splitting the window from left to right.  To create a vertical split, you can use `:vsp` or `:vsplit` to create a split of the same file/ blank file.  If you already have a file open, it will open the same file in the split as long as you don't specify which file to open. You can specify the name of the file after the command `:vsp filename` or `:vsplit filename`  It's not like that you can create only a single split, you can create multiple vertical splits. That can get pretty wild pretty quickly.  In the above screenshot, I have created 5 vertical splits from a single window, so making them equally wide and evenly spaced. This might not be useful every time but can get quite handy in some tricky situations. ### Horizontal Splits Similar to Vertical splits, we have horizontal Splits indicating to split from top to bottom. We can **split a single window into two halves horizontally** or a **sleeping split between the windows**. The below image clearly shows a horizontal split between two windows. Here we are splitting a single window into two windows. We can also think it of in splitting the window from top to bottom.  To create a horizontal split, you can use `:sp` or `:split` to create a horizontal split of the same file/ blank file. This will create a blank file inside a horizontal split.  Similar to the vertical splits, you can open files by creating the split. You can use the command `:sp filename` or `:split filename` to create the horizontal split between the windows and opening a specified file in it.  Again as seen in the vertical split, we can create as many splits as we like. This looks very ugly but who knows when you may need this. We can basically create Splits in any order of vertical or horizontal, we'll see it in the later section to the same.  ### Vertical and Horizontal Splits Together We can also create Vertical split and horizontal splits independently like we are not restricted to only creating the only kinds of splits at a time. You can create a vertical split once and then can create a horizontal split within the vertical split created, in this way we can make good use of both the splits as per our needs.   ## Moving around Splits Now let's talk about the navigation around these splits, it's again a simple thing. We can use `Ctrl+w` as our primary command along with Vim navigation to move around the splits. Like for example `Ctrl + w + h` will take you to the left split to your current open window. You can also use `Ctrl+w`** twice** to hop around to the next window split in a repeating loop. So, we can use: `Ctrl + w `+ `w` -> Switch to the next split (in the order of creation of those splits). `Ctrl + w ` + `h` -> Move to the left split. `Ctrl + w ` + `l` -> Move to the right split. `Ctrl + w ` + `j` -> Move to the below split. `Ctrl + w ` + `k` -> Move to a upper split. These commands might be good enough to pull you through any splits from anywhere, it just becomes easy to use Vim navigation inside of these. You can use arrow keys if you are stuck somewhere but using Vim key bindings will work out of the box. ## Creating Splits using Keyboard shortcuts If you think you are wasting time going into the command mode and typing the commands to create splits, well there are some shortcuts for you. `Ctrl + w` + `v` -> Create a vertical split. `Ctrl + w `+ `s` -> Create a horizontal split. This will open the split with the current file in the original window, so if you need to change the file, you can use the edit command (`:e filename`) inside the split. ## Rearranging the Window Splits If you have a specific set of splits of a kind open, you can rotate between those. Like for example, if we have a horizontal split, you can rotate the split to move the upper split down and below split up.  From the above illustration, we can see we rotated(swapped) the splits. We can also use certain commands to arrange the splits into appropriate positions according to the user. `Ctrl + w` + `r` -> Swap the two splits(either horizontal or vertical) `Ctrl + w` + `H` -> Move the split to the left ( `Ctrl + w` + `Shift + h`). `Ctrl + w` + `J` -> Move the split down ( `Ctrl + w` + `Shift + j`). `Ctrl + w` + `K` -> Move the split up ( `Ctrl + w` + `Shift + k`). . `Ctrl + w` + `L` -> Move the split to the right ( `Ctrl + w` + `Shift + l`).  We can see that we were able to swap the splits to a location suitable according to our wish. We can definitely switch the splits internally as well. We can resize the splits as per the requirement and remove the equality in those splits. We have a couple of options to do this: `Ctrl +w` + `+` -> Increase the height of the current split. `Ctrl +w` + `-` -> Decrease the height of the current split. `Ctrl +w` + `>` -> Increase the width of the current split. `Ctrl +w` + `<` -> Decrease the width of the current split. `Ctrl +w` + `=` -> Make the splits equal in width and height.  We can also use `:resize {number}` to manually set the size of the horizontal split and `:vertical resize {number}` to manually set the size of the vertical split. This is really risky, like if you know what you are doing then it's totally fine. The commands demonstrated earlier are really adjustable and user-friendly. We can also use `Ctrl + w` + `_` to minimize all the window split except the current one. ## Closing the Splits Now after doing all sorts of wizardry with the window splits the finishing touch is to close those splits after use. We can surely use `:q` to close the current window split but that is not intuitive to enter each split and manually close all of them, we can use: `Ctrl +w` + `c` -> Close the current split. `Ctrl +w` + `o` -> Quit all other splits except the current one.  ## Conclusion So, we have seen the basics of using Window Splits in Vim. Please let me know if there is some important point that is missing. Thank you for reading through here. We have seen how to create, navigate, rearrange, closing and other basic stuff related to Window splits in Vim. ### References: - [Sodocumentation](https://sodocumentation.net/vim/topic/1705/split-windows) - [Linux Hint Vim Split Screen](https://linuxhint.com/vim_split_screen/) - [Starefossen](https://gist.github.com/Starefossen/5957088) Hopefully, this might have given you some good idea to deal with Vim in windows splits. Happy Coding and Viming :) https://www.meetgor.com/posts/vim-tabs Introduction So, you are playing with Vim and getting good at but something pulls you off. Basically can't get used to switching between files or windows, and t Tue, 03 Aug 2021 00:00:00 UTC ## Introduction So, you are playing with Vim and getting good at but something pulls you off. Basically can't get used to switching between files or windows, and that's totally fine. Some may prefer using Window splitting, file managers, and whatnot. But let me introduce you to TABS in Vim. A simple and elegant way to open multiple files in Vim. ## Opening Tabs To open a tab, you can press `:tabnew` or `:tabedit` to open a blank Tab with no file open in it. This basically works like the `:e` command, which opens a buffer for you with no named file. If you already have an existing file in the current folder you are in, then you can press `:tabf filename` or `:tabnew filename` or `:tabedit filename`. This also applies to opening folders or directories, which will open the file structure in Vim buffer.  From the above illustration, we can see that the new tab was created using the filename with the `tabf` command and an empty tab was created with `tabnew` command. Ya, we can use `tabnew` for both cases but it saves time to write two more letters. It depends on the preference as you don't have to remember one more command in this case. You can also customize the commands if you feel they are too big to type in like a simple mapping would do the trick for opening the tabs for you. ### To open a Tab with a file specified. - `:tabf filename` - `:tabnew filename` - `:tabedit filename` ### Open a Tab without any file specified. - `:tabnew` - `:tabedit` You can open the tabs as per your choice like it could be ideal if you are gonna use certain files for a longer duration of time. This could be very ideal for various programming cases especially in Web, Android, Application Development where we need to edit a few files again and again. If you prefer Window-Splitting, that's totally fine, this is just to tell that there exist other ways as well. ## Switching Tabs Now if you are comfortable with opening tabs, we can now move on to switching between tabs. If you just have few tabs open, you can easily switch to the next tab using `gt` and to the previous tab using `gT` commands. But if you are in a great mode and want to open ten-twenty tabs XD, then you can use the numbers before the `gt` command. Like you can type `5gt` to move to the 5th Tab. If you do not know which tab is which, you can type in `:tabs` and this will open up the currently open tabs along with the numbers.  You can see my keystrokes(except the last keystroke) in the lower right corner. We can easily switch between tabs using the three sets of commands and surely configure them as per your preference. We also saw the `:tabs` command which can be pretty handy if you are working with a number of tabs. You can move around Tabs using some special commands like `:tablast` to move to the last tab and `:tabfirst` to move to the first tab. - `gt` -> Move to the **NEXT** Tab being in Normal mode. - `gT` -> Move to the **PREVIOUS** Tab being in Normal mode. - `ngt` -> Move to the Nth Tab in Normal mode. (n is any number of tab which are opened). - `:tablast` -> Move to the **LAST** Tab. - `:tabfirst` -> Move to the **FIRST** Tab. - `tabs` -> Get a list of Tabs which are currently opened. (includes file opened in the tab and the number) ## Closing Tabs So, after opening tabs you want to close em right? That is quite simple as expected. Just type `:tabc`, this will delete the current tab. You can close the specific tab by prefixing `tabc` with the number of that tab. Like if you want to delete the 2nd tab, use `:2tabc` to close the 2nd tab. If you want to reopen the closed tab, you can look out for the buffer name using `:ls` and then after finding the number of buffers in which your tab was open, you can type `:tabnew +nbuf`, here n is that number of the buffer. If you want to close all the tabs except the current one, you can use `:tabo`. This will clear the tabs except in which you are in, hence it will also collapse the top tab bar showing the file opened in those tabs.  As from the above GIF, we can see we located the number of the buffer last closed as we knew the name of the file which was opened in that tab. We also saw how to delete the specific tab using its number and the current tab. - `:tabc` -> Close the current tab. - `:ntabc`-> Close the Nth Tab. - `:tabo` -> Close all the tabs except the current Tab. ## Re-ordering Tabs This is a very tiny little detail but becomes a super tool in many cases. Let's say you want some reference of some content in the file, again and again, it's quite likely you should make the tabs nearby instead of switching tabs again and again. You can use Window splitting in this case, though we will see how to reorder tabs just for having the grasp on using Tabs in Vim. To reorder tabs, you are basically moving a tab from one position to other. Let's say you have a Tab at position `5` which is your current tab, you want it at position 2. So what you can do is move the current tab to position two, as simple as to speak `:tabm 1`. This will move the current tab which is at number 5 to the 2nd position. Remember the tab order is 0 based so just use the number you are thinking minus 1. So the command becomes `:tabm n`, where n is the index of the tab(starts from 0, the left-most tab). If you want to move to the last tab, you would not specify any number just type the `tabm` command, and that's it.  From the above example, we were able to move around the tabs to our desired location without a hassle. This is some basic stuff you can do with Tabs in Vim, surely you can add in your custom mappings to enhance the productivity and improve the workflow in Tabs in Vim. - `:tabm n` -> Move the current opened Tab to the Nth position (Starts from 0). ## Conclusion So, we have seen how we can use Tabs and move around in between files and folders, we are now able to open, close, move, navigate around the tabs in Vim. By using some custom mappings, this can be overhauled for much fewer keystrokes that get in it. There are many other navigation techniques in VIm, and using Tabs is one of them, surely it won't suit everyone but there will be someone who will prefer using this. Thank you for reading till here. I hope you learned something from this to enhance your grasp in Vim. Happy Coding and Viming :) https://www.meetgor.com/posts/bash-dictionary-scrapper Introduction Web Scraping is quite an interesting and powerful tool or skill to have in a Programmer's toolkit. It helps in analyzing data and getting some info Tue, 27 Jul 2021 00:00:00 UTC ## Introduction Web Scraping is quite an interesting and powerful tool or skill to have in a Programmer's toolkit. It helps in analyzing data and getting some information in various formats. Web Scraping is a process in which a user fetches a website's content using some pattern in those HTML tags and the desired content to be fetched or scraped. For this article, we aim to fetch the meaning of a word entered by the user from the dictionary.com website. We need to print just the meaning of the word from the HTML tags in it. We must have a good understanding of HTML and some basic Linux tools such as cURL, grep, sed, and others for doing all of these.  ## Inspecting the Target Website To begin with, scrapping the website, first, it is absolutely important to inspect the website and view its source code. For that, we can make use of Inspect tool in our Browsers. Just Right-click on the website you are viewing or the website for scraping, a list of options appears in front of you. You have to select Inspect option( also Shift + Ctrl + I), this will open a side window with a plethora of options. You simply have to select Elements from the top of the menus. The code that you will see is the source code of the website. No, don't think you can change the content of the website from here :)  Inspect Tool in the browser. Now we have to analyze the website with the content which we want to scrape. You can go on for clicking the `select the element in the page to inspect it` option or icon in the top left-hand side corner. This will allow you to inspect the particular element that you selected on the webpage. You can now see the element tag, id, class, and other attributes required to fetch the element's content. ## Selecting the particular element from the website to view the source code. ### Accessing the website from the Command line/terminal Now the website structure is being understood we can actually move to scrap it. For that, we need to have the web site's content on our local machine. First of all, we need to access the website from elsewhere not from the browser, because you cannot copy-paste content from there. So let's use Command Line here. We have a popular tool known as `cURL`, which stands for client URL. The tool fetches the contents of the provided URL. It also has several parameters or arguments that can be used to modify its output. We can use the command ``` curl -o meaning.txt https://www.dictionary.com/browse/computer# ``` The above command fetches the HTML page for the word Computer, it could be any word you might be searching for. ### Understanding the Website Structure. Here comes the time to explain the structure of dictionary.com. When you search a word on the website(dictionary.com), you are routed to `/browse` which then fetches the word for you and defaults you to the `/browse/word#` (the word can be any word you searched). The curl command dumps the output in the `meaning.txt` or any specified file. If you see the contents of the file, it is the same as on the web. So we are going to store the meaning of the searched word in the meaning.txt file, you can customize the name and command however you like. Voila! you successfully scraped a webpage. Now the next target is to filter the webpage content. ### Filtering Content from Website local file Now we have the content of the webpage on our local machine, we need to search or filter out the useful content and remove the unwanted tags and elements. For that, we can use commands such as `grep` and `sed`. ### Finding Tags to Extract content. We need to find patterns and similarities in the tags that contain the text of the meaning of the specified word. From the analysis of the webpage, we see that the element `<span class="one-click-content css-nnyc96 e1q3nk1v1">` contains the actual meaning. We just need the basic meaning, we may not need examples and long lengthy definitions on our Terminal, So we will go with filtering out the span tag with a class called `one-click-content css-nnyc96 e1q3nk1v1`. To do that we can use the grep command, which can print the text or line matching the specified expression or text. Here we need the span element with the particular class name so we will use regular expressions to find it more effectively. ```shell grep -oP '(?<=<span class="one-click-content css-nnyc96 e1q3nk1v1">).*?(?=</span>)' meaning.txt >temp.txt ``` ### Using GREP command to filter. The above command will search and return only lines that are contained in the span tags with that particular class name from the meaning.txt file which we appended to fill the webpage's source code. The `-oP` are the arguments that return Only the matching cases and `-P` the coming expression is a Perl Regex. The command will return everything in between those tags. Finally, we are storing the result or output in `temp.txt`. Now, if you think we are done, then it's wrong, the webpage can have internal or external links embedded inside of the elements as well, so we need to again filter out the HTML tags from the `temp.txt` file. For that, we will introduce another tool to filter text called `sed` or Stream editor. This tool allows us to filter the stream field and print or store the outcome. The following code will remove the HTML tags from the scrapped text. ### Using SED command to remove embedded ```shell sed -i 's/<[^>]*>//g' temp.txt >meaning.txt ``` The above command filters the text and removes the HTML tags from the `temp.txt `file using regular expressions. The `-i` command allows us to store the output in a file `meaning.txt`. We have used Regex to remove `<>` tags from the file and hence anything in between these is also removed and we get the only pure text but it may also contain special characters and symbols. To remove that we'll again use `grep` and filter the fine meaning in our file. ### Removing Special Characters from the Content using GREP commands. ```shell grep -v '^\s*$\|^\s*\#' temp.txt >meaning.txt ``` Now from the above command removes the special characters such as `$,#`, and others from the temp.txt file. We finally store everything filtered in the meaning.txt file. If you understood till here, the next concrete step will be super easy for you, as we will assemble everything here in a shell script. ## Making the Shell Script ```bash #!/bin/bash read -p "Enter the word to find meaning : " word output="meaning.txt" url="https://www.dictionary.com/browse/$word#" curl -o $output $url clear grep -oP '(?<=<span class="one-click-content css-nnyc96 e1q3nk1v1">).*?(?=</span>)' $output >temp.txt sed -i 's/<[^>]*>//g' temp.txt >$output grep -v '^\s*$\|^\s*\#' temp.txt >$output echo "$word" while read meaning do echo $meaning done < $output ``` We can clearly see most of the commands are the same, but some have been modified to avoid repetition and automation. Firstly, I have taken user input of word from the user and stored it in with an appropriate variable name. Next, I have created another variable to store the file name in which we are going to store the meaning of the word, Also a variable for the URL of the website we are searching for. We have used a variable to access the required URL. Then we invoke `cURL` to the file which we want to store using the variable we created and the URL variable So creating variables makes our script quite easy to manage and also it improves the readability of the script. ## Updating cURL command We can also update the curl command by adding `"&> /dev/null"` this will dump the curl output of network analysis. So we will only get the output of the meaning.txt file. It is optional to add the following into your code as it depends on the operating system so we can optionally use clear command to wipe out the curl output. ## Printing the output file line by line. To print the meaning in the output file, we need to print each line separately as the meanings are distinct. Therefore, we will use a while loop with the output file and echo the line variable we have used as the loop iterator. ## Script Screenshots:     ## Output Conclusion From the above output, we have scrapped the meaning of the word `Mathematics`, `code`, and `python`. It works only for the words which are on the dictionary.com website. We have successfully made a scrapper that scraps the meaning of the input word from the dictionary.com website, ## Appropriate use of Web-Scrapping. We must be careful and not scrape any website without reading its privacy policy. If they allow scraping from their website, then only you should scrape the content and not use it for any monetization of the content. This was just used for demonstrating some idea about web scrapping using BASH and just meant for teaching purposes. Therefore, it is quite easy to scrape the website's content especially if you find any patterns in the code structure. We were able to make a script that can print the meaning of the input word from the base of the website dictionary.com. We can see how Bash can be powerful in terms of web scrapping. I hope you found this interesting and inspiring. Thank you for reading. Happy Coding :) https://www.meetgor.com/posts/vim-registers Introduction Have you ever found it difficult to manage and operate text in Vim, especially in Cut/Copy/Paste stuff. You cannot figure out how on earth am I sup Wed, 21 Jul 2021 00:00:00 UTC ## Introduction Have you ever found it difficult to manage and operate text in Vim, especially in Cut/Copy/Paste stuff. You cannot figure out how on earth am I supposed to retrieve my un-saved, un-committed changes in Vim. Then this my dear friend is an article for you. You will learn what are Vim registers and where your deletes and copied stuff resides. This won't be a comprehensive guide but enough to let you through in performing the wizardry of registers in Vim. ## What are Vim Registers? Vim Registers are memory addresses that store data. Technically they are spaces of memory that Vim can store to retrieve later. It can store text, operations, commands, and anything you can do with text in Vim. ## Access Vim Register To store something in register `x` we can use the format `"[register-name][command]`. This will perform the command and store the output in the register specified. A register can be accessed by using the double quotes `"` before its name. Let's say we have stored something in x register we can access its content using "x in the normal mode and lastly we can perform commands here.  For example, in the above gif, I tanked the selected text using the yank command but I stored it in the register x by prefixing the command with `"x`. Hence the command became `"xy` and similarly for pasting from the x register we can use `"xp`, quite simple, right? But what is the purpose of it? This can really shine in handling multiple files and imagine you lost your yanked text by using yank somewhere else. Vim registers add extra dimensions in storing and retrieving text, it is not used often but can shine really well in certain tricky situations. ## Commonly used Vim Registers You will use some Registers daily in Vim if you use it all the time. Some common ones are the clipboard registers/ selection registers `"+` and `"*`. These are used to perform operations on the content stored in the system clipboard. They come quite handy in copy and paste in Vim from other applications such as Stack Overflow XD. Just kidding, it can be used in any other application on the system. We can add `p` or `y` after the registers to paste and respectively in Vim, so to paste and yank text to and from Vim, you can use `"+p` and `"+y`. ## Types of Vim Registers There are many different types of vim registers used for particular needs and also some for custom editing. - ### 26 Named registers (`a` to `z`) These are the registers just for you. You can use this for whatever copy/pasting/editing and other creative stuff you could do with this. In the example where we utilized the `x` register, which was one of them. We can use a to z any register we want, it just boils down to the ease of the key combination as per preference. **Remember anything you yank into any register will also be stored in the unnamed register (discussed later)**. Also, if you try to use capital named registers (A-Z), the content will be appended to the lowercase named registers as well along with the unnamed register. Making it a bit safer to work with but messy at the same time. - ### 10 numbered registers (`0` to `9`) These are the registers updated automatically filled in by Vim, they store the recently yanked or deleted content in the current buffer. - The `0` register stores the most recent yanked/copied text. - The `1` register stores the most recent deletion of text. - The `2` register stores the 2nd most recent deletion. .. and so on. The content of the 9th register is flushed away every single time you delete something. The contents are shifted to the next numbered registers every time there is a deletion. For example, If we delete something the content is stored in the `1` register but its previous content gets shifted to the `2` register, and so on. So we have 9 clipboards for our deletion history. That is undoubtedly a powerful thing. - ### Selection and drop registers (`*` , `+` , and `~`) As discussed earlier, Selection registers are used for storing the contents of the system clipboard. There are differences in the `*` and `+` for Unix-like operating systems but they can do similar things. In Unix-like Operating systems, the `*` register stores the text from the mouse selection in the X Window, whereas the `+` register is used to store the text system clipboard. On Windows and other operating systems besides Unix-like, both of them function similarly. The Drop register `~` is available only for the Gvim version as it stores the text from the last drag and drops/drag down operations performed. - ### Read-only registers (`:`, `.`, and `%`) The registers `:`, `.`, and `%` are Read-only, which means you cannot change their content, they are modified and maintained by VIm automatically. Using the `:` register, we can get the content of any register in the Insert mode so that we can copy the contents and edit if we have stored it in some registers. This is quite a neat little feature, not used most of the time but it can become the most powerful tool in corner cases and tricky situations.  From the example in the gif, we deleted the line, and hence the content was stored in register 1 so when we press `Ctrl + R` in Insert mode, we can now paste in the contents of any register. Here we used the register `1` to get our content back. The `%` register holds the name of the current file, this is really a great and life-saving editor, as it is also used in automating the compilation and running of the source files from Vim itself. The `.` register contains the last inserted text, this can also be used in various scenarios. They make Vim a bit polished and well organized.  From the above gif, we inserted some text, and using the Read-only register, we inserted the last inserted text to the screen using the `.` register and also we used the register `%` to get the current file name. - ### Expression and Search pattern register (`=`, `/`) The register `=` is used to perform operations and store them in the register. Vim has some built-in functions and some of them are listed [here](https://renenyffenegger.ch/notes/development/vim/script/vimscript/functions/index). We can use these functions after we type `Ctrl + R` in insert mode and `=` to assign it to the register `=`. I have used some simple multiplication operations in the above gif, but you can take this further however you like and desire. The register `/` takes searching in Vim even further. We use /pattern in normal mode to search for the pattern. But the actual work is done with help of the register. We actually store the text in the register and Vim accesses it later t find the next pattern or any other operations performed with the pattern. This is again a read-only register but you can change its content using some methods and practices. It is not preferred to change the read-only registers as it can mess up the functions of Vim. - ### small delete and the alternate file register (`-`, `#`) Small delete is quite a meaningful and sensible word to put in the context of its functioning. The `-` register or small delete register stores the content of some small deletes like characters or words. Basically, anything smaller than a line is small for Vim. If you delete something let's say a word from a line, it'll be stored in the `-` register. The '#' register or alternate file register is the register that stores the alternate file in Vim. An alternate file is a file edited before the current file in Vim. If you edited a `file-x` and now you are in the `file-y`, the `file-x` becomes your alternate file.  This example shows how we can use the small delete register to retrieve back the small deletions. Also, we saw the current file and alternate files in Vim and in which registers they are stored. - ### Unnamed and black hole register (`"`, `_`) Unnamed register `"`, simply to put in it is a register which stores everything from deletions to yank. It stores the text performed by the d, D, x, X, s, S, c, C, y, Y, some variants of these commands. This is quite good and a bad feature sometimes. It is nice to have such types of registers in hand. Wear your space goggles as we see will how Black holes work :) Blackhole register is an elegant touch to the series of registers we have. The register acts as a sink for our deletions. We can simply dump anything we don't care about in this register. This register helps in scenarios where we want to avoid adding stuff from deletions to the named or numbered registers. Isn't it like Blackholes? Just Swallow the things.  The above example makes it clear that we simply dump the deletion to the `_` register and avoid changing the rest of the registers which may replace the deletion and lose the saved configuration or set up of registers. We first deleted the line `solve(s);` and dumped the statement `string s = "0110` into the black hole register `_`, when we retrieve with thee unnamed register `"` we get back the deleted statement rather than the dumped statement. Quite neat and elegant. ## Where can you use Vim registers Well, this may differ as per the experience with your programming journey, you may find it useless as a beginner but mind you this is some extraordinary stuff that can boost some development time. You may be tempted to use it the scenarios where there is a huge depth in file systems and the code to be edited is a lot for you. When you have a ton of things to be edited systematically, this concept can add lights and dimensions to your work and make it comfortable and risk-free. ## Conclusion  The above chart is quite simple to remember if you are really into making Vim for complex projects and even for staying productive in Vim editor. Hope you find it interesting and easy to remember. This was a small introduction to Vim-registers but is a small topic with extensive usage and skill-dependent concept. Vim is already a customizable editor, registers add an extra dimension to its customizability in editing and programming. Thank you for listening to me. Happy Coding :) **References** : [baeldung.com](https://www.baeldung.com/linux/vim-registers), [brianstorti.com](https://www.brianstorti.com/vim-registers/) https://www.meetgor.com/posts/vim-editing-speed Introduction Editing/ Writing is a crucial part of any text editor, the swiftness with which one can make changes in a file(s) or a structure is a bit dependent Sun, 18 Jul 2021 00:00:00 UTC ## Introduction Editing/ Writing is a crucial part of any text editor, the swiftness with which one can make changes in a file(s) or a structure is a bit dependent on the editor we use as well. Vim can be a bit hassle for beginners in the initial stage but it becomes second nature soon. It also depends majorly on the level of usage and the area of usage. If you are an advanced programmer, you will use these commands extensively, and might not be a big deal for you, But for a beginner, it might take some time to get used to the workflow in Vim. This article won't be covering the basics of editing in Vim but the ways to save time on the basic level commands to improve efficiency. That being said, beginners can also read to be aware of the possibilities and tricks in Vim. Below are some quick basic commands for getting started in editing and improving the overall workflow. ### Entering in into the Insert Mode `i` -> Enter into **Insert mode** from the cursor. `I` -> Enter into **Insert mode** from the beginning of the current line. `a` -> Enter into **Insert mode** just after the cursor. `A` -> Enter into **Insert mode** from the end of the current line. `o` -> Enter into **Insert mode** below the current line. `O` -> Enter the **Insert mode** above the current line. The above commands are purely to add text in the line or above or below the line without deleting anything. The following commands will delete some characters/words/lines and then entering into the Insert mode. `s` -> delete the character under the cursor and enter into **Insert mode**. `cw` -> Delete the word under cursor and enter into **Insert mode**. `S` -> delete the entire line under the cursor and enter into **Insert mode**. ## Cut Copy Paste Commands This set of commands are quite helpful as a programmer and it is used quite frequently. These commands can surely boost the time to cut-copy-paste but also they will provide more customization to the way you do these tasks. ### Cut/Delete Commands The below-mentioned commands can be used in Normal or any Visual/ Selection mode as it depends whether you have selected the text or you want to work in Normal mode itself. `dd` -> Delete the current entire line. `dw` -> Delete the word on the cursor. `d0` -> Deletes the line from the current cursor position to the beginning of the line. `D` or `d$` -> Deletes the line from the current cursor position to the end of the line. `d` -> Delete the selected text (only in Visual/Visual-Line/Visual-Block/Selection mode). `x` -> Delete the character under the cursor. ### Yank/Copy Commands The following commands are used in the Normal mode as they perform the copying of text with words and lines only. `yw` -> yank(copy) the word on the cursor. `y$` -> yank(copy) line till the end from the current cursor position. `yy` -> yank(copy) the current entire line to the unnamed register (""). You have to be in Visual/ Visual line/ Selection mode to yank the text for the next set of yanking commands. `y` -> yank(copy) the selected text to the unnamed register (""). `"+y` -> yank(copy) the selected text to the system clipboard ("+ register). ### Paste Commands `p` -> Paste the content of the unnamed register("") below the cursor. `P` -> Paste the content of the unnamed register("") above the cursor. `"+p` -> Paster the content of system clipboard ("+ register) to the cursor. ### Replacing Text Replacing is a great option for instant productivity, if you want to make some minor changes, you don't have to go into insert mode and delete and then edit the text. Instead, the replace commands such as `r` and `R` allow us to replace some characters being in Normal and Replace mode respectively. This can be used very heavily if you just want to replace it instead of adding/removing text. `r` -> replace the character under the cursor with the following key entered from the keyboard. `R` -> Enter into **Replace mode**( replace the character with the specified word from the keyboard). ### Undoing and Redoing We often make mistakes and want to revert to the changes we have to make and start from the last save. The following sets of commands will make us do exactly that. `u` -> Undo the last made changes before saving. `U` -> Restore the changes in the entire line. `<C-R>` -> Redo the last undo (un-undo -> revert back changes). ### Search and Replacement Some of the below-mentioned commands are a great set for bulk replacement and addition. We can either replace a particular pattern in the entire file or in specific parts of the file as mentioned and explained as follows: `:%s/old/new` -> Replace the word `old` with `new` in the entire file. `:4s/old/new` -> Replace the word `old` with the word `new` on line 4( where 4 can be any number of lines in the file). `:2, 9s/old/new` -> Replace the word `old` with `new` between the lines 2 and 9 inclusive(where 2 and 9 can be any number of lines in the file). `:%s/^/This` -> Add `This` to the beginning of each line in the file. `:%s/$/That` -> Append the word `That` to the end of each file. You can notice that the `%` symbol here indicates the entire file. We can skip prefixing s with `%` to make changes only in the current line or any number of lines specified instead of it. This command is quite a lot customizable and powerful, I can't show each and every combination of this command. It should be used as per requirement and thus should be modified accordingly. ### Indenting Text Indenting is quite important in certain languages like Python, YAML, Haskell, etc. This can get really frustrating if you even miss a single indentation, you have to format everything to a proper indentation scratch. But thanks to powerful Text-editors and IDEs which have made the indentation quite easy and even auto-correct the wrong indentation. Vim has some commands to make those indentations much easier and it also has a customizable number of spaces in its config file called vimrc. `>>` -> Indent or shift the current line to the right. (normal mode) `<<` -> Unindent shift the current line to the left. (normal mode) `>` -> Indent or shift the selected text to right. (Visual/Visual-line/VIsual-block/Select mode) `<` -> Unindent or shift the selected text to left. (Visual/Visual-line/VIsual-block/Select mode) If your file is saved as a particular language that supports indentation, it will automatically indent lines for you but it cannot be reliable. So, we need to [IndentLine](https://github.com/Yggdroot/indentLine), and others as well. ## Miscellaneous `~` -> Convert the character under the cursor to upper case/ lower case. `vip` -> Yank a entire paragraph ( till empty line). `gu` -> Convert the selected text into lowercase. (Visual/Select Mode) `gU` -> Convert the selected text into Uppercase. (Visual/Select Mode) All of the above commands were somewhat basic and commonly used but if used along with other key shortcuts for movement can also improve the editing speed quite considerably. Just keep using these commands and you'll be amazed by the speed you've developed. Thanks for reading. Happy Coding :) References: [catswhocode](https://catswhocode.com/vim-commands/) [The valuable dev](https://thevaluable.dev/vim-advanced/) [rtorr.com](https://vim.rtorr.com/) https://www.meetgor.com/posts/bash-crypto-scrapper Introduction Cryptocurrency is in such a hype that it is attracting even nerds and terminal fanboys, it is quite comfortable for such people to view the price i Thu, 15 Jul 2021 00:00:00 UTC ## Introduction Cryptocurrency is in such a hype that it is attracting even nerds and terminal fanboys, it is quite comfortable for such people to view the price inside of their terminal, also for normal people to learn about how to interact with an API from the local machine. We are gonna make a script about 20 lines in BASH to extract data from the [coingecko cryptocurrency API](https://www.coingecko.com/en/api#explore-api) and some tools such as grep and sed. That being said let's start scripting.  This is how the script will work like. It is not a fancy script just some basic commands to extract data from the API. ## Getting familiar with the API This is quite an important step as this will decide what type of data we will get from it. So head out to [coingecko cryptocurrency API](https://www.coingecko.com/en/api#explore-api) and start exploring some things yourselves. I am gonna cover everything you need for the script but just for your knowledge and if you are really into crypto.  Hopefully, you should see the following screen and after clicking on "Get Started", you will see a list of the API endpoints. After playing with it for several hours, I found the perfect and precise endpoint to achieve our target. Many o them work well but the problem with many of them was the precision of the price, there it could not show a reliable price for some low-valued coin/tokens. There are tons of options to choose from, but every option has its own advantage and disadvantage. Based on my tiny experience, the best fit for the required task will be the "Get historical market data to include price, market cap, and 24h volume (granularity auto)" option in the coin section. It will give the price with very nice precision and also in any national currency as well. So when you find the option, which looks like this:  If you click on that option you will get a button to **"Try it out"**, just click on it and now you can parse data to the API and respond with the JSON response file like the text below as output.  ### Filling in the coin id You need the **id**which is the code of the cryptocurrency on coingecko so a few ids of famous cryptocurrencies are: `bitcoin`, `ethereum`, `dogecoin`, `shiba-inu`, `basic-attention-token`, and if you want more coins you can check the coingecko API coins list [here](https://api.coingecko.com/api/v3/coins/list) and now you can search using the shortcut F3 and type the name of the coin you require. ### Filling in the currency We need a currency to display the price of the selected coin in terms of a particular currency or its equivalent price in the currency. Every national currency has a code attached to it, for example, USD is the code for US Dollar, INR for Indian Rupee, EUR for Euro, AUD for Australian Dollar, and so on. You can find the list [here](https://www.iban.com/currency-codes). ### The number of days We also need the number of days to indicate the price of the coin before the provided day, but we want today's and latest price, so we can enter the number of days as 0. You can get the price as your wish but we want the current price so we are using 0. So your request should look somewhat like this, just for particular coin pair:  After filling in with your favorite coin-currency pair, you can now click on **"Execute"**. You will get a list of prices, volume, market cap, and all of that stuff. It should look somewhat like this:  If you look carefully and are aware of the current price, the second item in the prices object is the actual price of ethereum at some point in time. We need to extract that using tools and utilities in BASH. They have also provided a cURL command to fetch the API, how nice of Coingecko :) So, that was the introduction to the Coingecko API, you can now explore various endpoints and what they respond to, and what are inputs they take. Many of them do not give precise enough price, so check out that as well. ## Fetching data from the API Now, we'll finally start coding. Firstly we'll need input from the user, the coin-id, currency code, and optionally the number of days. We will read the input by using the read command and providing the prompt argument. ```bash #!/bin/bash read -p "Enter the coin name: " coin read -p "Enter your national currency: " crncy days=0 #read -p "Enter the number of days before today to get its price: " days ``` I am not inputting the number of days before the equivalent price as it's not our mission, but if have commented on the part if you want to get the price before some days. The day value is hardcoded to 0 meaning the current day. Now we need the golden piece of this script, the cURL command. The command is just pre-written for us XD. But hey we need to modify it a bit. We need to store the output of the cURL command in a fie for modifying it further. We can use piping the command and avoid using files but that would make the command quite big. So, I just use the files to handle and trim the output according to what I need. This is the default command that coingecko gave us: ```bash #!/bin/bash curl -X 'GET' \ 'https://api.coingecko.com/api/v3/coins/ethereum/market_chart?vs_currency=usd&days=0' \ -H 'accept: application/json' ``` We send a GET request to the API endpoint with the provide URL and we accept the response in the form of JSON. The -X is for providing the option of either GET, POST, or any other API interaction code. But this is a REST API and we do not want to mess up the database by posting, deleting, or updating anything just to GET the data from the API. We use -H to pass the URL to the head and the type of response in this case JSON. We'll change the default command to this, ```bash curl -o temp.json -X 'GET' \ 'https://api.coingecko.com/api/v3/coins/'$coin'/market_chart?vs_currency='$crncy'&days='$days'' \ -H 'accept: application/json' &> /dev/null ``` We need to modify the URL a bit to make it dynamic. The ethereum or any coin name should be dynamic, the currency should be dynamic, so we will use the variables created before to use now. We will use `'$variable'` in between the URL to embed the variable value in it. We change the ethereum or any coin name with `'$coin'` and the currency name with `'$crncy'` and the same for the days as well. We have to store the output in the temp.json file, we use -o to output the result in the specified file in the cURL command. It's optional to add `&> /dev/null` because it just flushes the output of cURL, it looks neater if we add it. And that is it, we have obtained the JSON response and stored it in a file, we have the access to the data and we just need to edit the file. ## Editing the response JSON file Now we start the actual editing and pattern finding in BASH. We have a plethora of tools to use like grep, sed, awk, and others, but I'll use only grep to keep things simple. Firstly we need to see the output( JSON file) again.  But the file stored locally has everything in a single line, like this:  So, we just need the numbers between `,` and `]],"market_caps"` right? OR The text between `,` and `]],"m`. It is quite simple to extract the required value, we will use grep and Pearl Regular Expression. ```bash grep -o -P '(?<=,).*(?=]],"m)' temp.json > price.txt ``` We are finding anything between the `,` and `]],"m` from the temp.json file and storing the output in the price.txt file. As simple to use and we have the current price of the coin in terms of the provided currency in the file price.txt. Now we have obtained the result in a crystal clear way, we need to store the price in a variable just for further usage and simplicity. We'll use a while loop that iterates over the file until it is End of the File. We'll store the value in the variable in the following way: ```bash while read val do p=$val done < price.txt ``` The value in the `p` variable. But we are not done yet, because if we see some values of certain coins which have quite low value, it displays in the scientific format. We'll tackle this in the next section. ## Converting the price from scientific notation to decimal If you try to print the values of coins with pretty low value like `shiba-inu` or `baby-doge-coin` or any other coin with less value then a penny. The value is expressed in scientific notation i.e like `1.998e-5` i.e `0.00001998`  This is not good-looking, is it? ```bash price=`printf "%.15f" $p` ``` The above command is quite similar to the C language. We are printing the value in the `p` variable with a precision of 15 decimal values, that is enough for any serious small value coin.  Now that was much better. That is it! How simple was that. Really BASH has some powerful commands and tools. ## Printing the price Finally, we need to print the output, and to keep it simple, we can print using the echo command. ```bash echo "The value of $coin in $crncy is = $price" ```  ## Script ```bash #!/bin/bash read -p "Enter the coin name : " coin read -p "Enter your national currency : " crncy days=0 #read -p "Enter the number of days past today: " days touch temp.json price.txt curl -o temp.json -X 'GET' \ 'https://api.coingecko.com/api/v3/coins/'$coin'/market_chart?vs_currency='$crncy'&days='$days'' \ -H 'accept: application/json' &> /dev/null grep -o -P '(?<=,).*(?=]],"m)' temp.json > price.txt while read val do p=$val done < price.txt price=`printf "%.15f" $p` echo "The value of $coin in $crncy is = $price" rm temp.json ``` That was it, I hope you liked it and learned something along with me. BASH truly has the potential to do a whole lot of stuff. This was just one of the many examples where we can do some data gathering or scraping from the web using various tools and utilities. Thank you for listening to me. Happy Coding:) https://www.meetgor.com/posts/bash-guide-p3 Bash or shell won't be much popular and powerful if they didn't have some tools and utilities baked in. But even further they are supported natively in Bash, wh Mon, 12 Jul 2021 00:00:00 UTC Bash or shell won't be much popular and powerful if they didn't have some tools and utilities baked in. But even further they are supported natively in Bash, which just makes every task and challenge quite hassle-free to deal with. In this part of the series, I'll try to cover an overview of some quite powerful and robust tools and utilities in Bash(shell in general) and also some of the advanced topics like dictionaries and positional parameters. Enough talk let's dive in. The topics to be covered in this part include the following: - Hash tables/dictionaries in BASH - Positional parameters - Aliases in BASH - Some Tools and utilities - grep/sed/awk - cat/tac/head/tail - cURL - find - bc - wc ## Bash dictionaries Bash dictionaries or hash tables are just like any other hash tables or keymaps in other programming languages. Bash dictionaries are quite similar to arrays but they have a key instead of the index(0,1,2...) and a value just like arrays. This can be quite useful for storing passwords with emails or usernames or any other way in which a value can be accessed only via a unique key. To declare a dictionary/ hash table, we can simply write `declare -A name`, this will declare an empty hash map for us. Further, we can populate the hash map with keys and values using the same syntax as of array just instead of numbers we can also have strings. ```bash #!/bin/bash declare -A fruits fruits["apple"]="red" fruits["mango"]="yellow" fruits["grapes"]="green" read -p "Enter the name of fruit : " name echo "The fruit is $name and its color is ${fruits[$name]} " ```  The above example depicts a way to declare, define and access the key values in a dictionary. The example may look silly but you get the idea. We can also access every key or value using the `@` variable and access the number of key-value pairs using the `#` variable just like an array. ```bash #!/bin/bash declare -A fruits fruits["apple"]="red" fruits["mango"]="yellow" fruits["grapes"]="green" for i in "${!fruits[@]}"; do echo "$i : ${fruits[$i]}" done echo "There are ${#fruits[@]} key-value pairs." ```  OK, this is very tricky they are the same variables but used slightly differently. Firstly in the range-based for loop `${!fruits[@]}`, focus on the `!` before the hash table name, this will expand(depict) the hash map's keys. This is used to access every key from the hash table and we can also see `#` at the beginning of the hash map name as it is used to represent the entire hash map further in the `{#fruits[@]}` we can also use `{#fruits[*]}`. Inside the for loop, `i` will be the key, and `{fruits[$i]}` will be the value for that `i` th key. Also, you can notice the bash interpreter automatically arranges the map in the alphabetical order of the values and not keys. This is quite a neat little feature that can come in handy a lot of times. If you want to delete or add any key-value pairs we can do that by the following commands: ```bash #!/bin/bash declare -A fruits fruits["apple"]="red" fruits["mango"]="yellow" fruits["grapes"]="green" for i in "${!fruits[@]}"; do echo "$i : ${fruits[$i]}" done echo "There are ${#fruits[@]} key-value pairs." unset fruits["mango"] echo "There are ${#fruits[@]} key-value pairs." fruits["strawberry"]="pink" for i in "${!fruits[@]}"; do echo "$i : ${fruits[$i]}" done echo "There are ${#fruits[@]} key-value pairs." ```  The above code example is a bit complex but easy enough to understand. We can delete the key-value pair using the unset command and pass in the key along with the hash map name. We can create a key-value pair by simple command as depicted in the above example. This was a basic overview of hash maps/dictionaries in BASH. ## Positional parameters We often use user input from within the script but there is another way to pass in parameters outside of the script using positional parameters. It basically allows us to pass in arguments or parameters from the command prompt/ shell and inside of the script, we can access them via Positional Parameters ( $1, $2, $3....$9, ${10} and so on). ```bash #!/bin/bash echo "first parameter : " $1 echo "second parameter : " $2 echo "eleventh parameter : " ${11} ```  You can see from the above script we have passed the parameters from the command line just after typing the filename. The positional parameter $0 is the file name and from 1 onwards the parameters are optional to run if only your script needs any parameters or input to work with. The numbers are just random and just used for demonstration. The 11th parameter or double-digit parameter starting from 10 onwards, you need to encapsulate the number in {curly braces}`${number}` because it won't interpret `$10` or any other number as just `$1` and print 0. So we can pass a list of parameters that should be space-separated. We can pass any relevant information such as a string, number, or file names as well. If we want to access all the parameters passed to the script, we can use `@` variable. You may know this symbol from the array section of part-II, it is used to access every element in the array. But here it is used to access every parameter passed to the script just behaving like a list of values. ```bash #!/bin/bash echo "The parameters passed are: " $@ ```  To get the number of the parameters passed to the script, we can use `#` variable. This is also a variable used in the array section for accessing the number of elements in the array, in this case, the number of parameters in the list. ```bash #!/bin/bash echo "The parameters passed are: " $# ```  Positional parameters allow to not take explicit input from the user from the script. This might not be used for the simple scripts but for administration purposes for the admins and users who know what does the script needs and it allows them to pass in arguments without designing the input system. ## Bash aliases Bash aliases are a great way of reducing the command length and making it much easier to type and work with the scripts or any development-related work. Alias is a file called bash_aliases inside the .bashrc folder that contains our shortcut commands, it has a particular order to map certain commands with others. Let's see what is an alias first and then we'll see how to set it up. ```bash alias cdc='cd C:/Users/acer/Desktop/New\ folder/Code/' ``` This will make it viable to just type cdc and I will be in this directory instead of printing all of the jargon. The command we need to use to replace the big command is `cdc`. The right command is the variable assigned the command and the left or its value is the command to be replaced with it.  This command will definitely defer on your machine and OS as the filesystems are different in each major operating system. We can quickly make other such alias or shortcuts so to speak for making the development process faster and efficient. Now let us see how we set up this environment for bash alias, it's quite straightforward. You need to create a hidden file named "bashrc" i.e the file name will be `.bashrc`. This file has to be in the root directory (the folder to which bash defaults). I do not mean the `root` directory in Linux but the repository in which your bash interpreter opens. Once you have created the file put any alias in the file and source the file using the command: ```bash source .bashrc ``` And that would do the trick, you can now test your macro or shortcut by opening a new instance of the terminal. If this doesn't work for you, then you can check [this article](https://opensource.com/article/19/7/bash-aliases) for a broader understanding of the setup. ## Bash tools & utilities What would you call BASH without grep or sed man! It's a sad life:( BASH comes with some absolutely powerful and handy tools and utilities such as grep, sed, awk, at, wc, find, tar, gzip, which, make, ping, cURL, wget, ssh, .... my words there is an unstoppable long list. Really they are quite important and lay the foundation for some quite complex tasks. Some web servers can become redundant if some of the tools went missing. Let us find why they are so powerful. ### grep GREP or global regular expression print is a tool or command that can find patterns using regular expressions in files/strings or any other piece of data. It's not just printing or searching for the text, besides all that it can also edit the file and store the output in the desired file or any variable by proving some arguments to it. Grep supports Pearl's regular expression as well. There is a lot of customization options and arguments available in grep that can just do anything. It becomes an irreplaceable tool for some complex tasks.  The above code finds the pattern "more text" in the file specified and prints the line to the screen, but we can modify the output we want, extract the output in a file and do all kinds of wizardry with this tool. This is just a basic, to get started example but trust me it's more than you think, this tool is used widely for web scrapping and pattern matching in quite a lot of use cases. ### sed SED or stream editor is another beast in BASH's toolkit, this is just a flawless tool. No words for this. This is a great tool but still underrated. This can actually edit the text inside the terminal, no graphical environment, no interface at all just commands, but it can do what a huge text editor can't! Save time, just edit text without opening anything except a terminal, becomes unbeatable in large files. This is surely a tiny little application that can skyrocket the efficiency development process.  The above example replaces the word `more` with `less` using the sed command if you type 3g or nth line preceded by g, you will replace the word on the nth line only. In this case, only`g` will replace every occurrence of the word. This is again a basic example of a sed command, its more if you go deeper, its more than a tool, its kind of a text-editor for wizards ;) ### awk awk or Aho, Weinberger, and Kernighan XD are the names of the developers of this application. This is another mind-blowing tool that can programmatically do a lot of stuff. This is like a programming language to a whole new level that can extrapolate and extract data from files and other forms of data. This is quite a great option if you want to quite neatly do something. It has great support libraries and functions that can even perform complex mathematical and scientific operations.  These are the topics for separate articles because it is insufficient to explain everything at once. ### cat / tac / head / tail CAT or concatenate is a tool used for printing out files, create files, sorting the contents of files, editing files, and a plethora of stuff. This command is generally used for printing the file but there is more to it like creating a file directly in the terminal, merging two files, and a ton of other operations. TAC or reverse of CAT is a tool used for everything that CAT can do but in reverse:) This is a wired tool but still quite useful sometimes. Head is a tool that will print or edit the text in the first 10 lines of the file, it can be used to extrapolate multiple files with similar content. Tail is a tool that will print or edit the text in the last 10 lines of the file, it can be used just like head but for the last few lines.  It turns out, you can not only print the first or last 10 lines but n lines by passing the -n as an argument, there is a ton of stuff to discover, this just drives me crazy. ### cURL cURL or client URL is a tool that can be used to transfer data via various network protocols. You might not believe but it is used in cars, televisions, routers, and other embedded systems for exchanging relevant data via appropriate protocols.  This example depicts how we can fetch data from an API using cURL and extract data in JSON format and use it for relevant tasks. This is again one of the best utility out there as it becomes quite remarkable and vintage. Despite being almost 30 years old, it shines bright in the tech world. ### find Find as the name suggests it is used to find files among the folders and directories in a file system. it becomes quite helpful in complex projects where the directory structure is deep and large.  The command `find *.txt` finds every txt file available on this directory. As simple as it can get. This is surely looking silly and idiotic but it finds its glory in large and complicated codebases. ### bc bc or basic calculator is a utility tool for performing mathematical and arithmetical operations in the terminal, this commands gets integrated with other commands such as awk really well and can be used for further extending the limits of what the command line development can do.  AWW! I could hear the excitement. That just added new dimensions into BASH. Just creativity and resonance to anything is the limit here. I am using [REPL.IT](http://repl.it/) here for using bash as I do not have it on my windows machine :( But that command is truly insane. ### wc wc or word count is a utility tool for counting and analyzing the size or count of characters, words, lines, or files in a given file structure. This is quite a handy tool for monitoring and keeping track of a system, also for general development purposes.  The above command prints out the word and lines in the provided file. This command `wc` can even compute the size of files and even more properties of files. Those were some of the quite powerful commands, tools, or utilities in BASH/shell. There are plenty of other commands not covered here because this an extremely large topic and even making separate articles or resources there will certainly and surely be some things that will get missed out, that's the beauty Linux or in general Computer Science has. Ok, that was a lot, but I hope you got some insights for learning more BASH or Linux in general. This is a wide topic and can't be covered entirely in a single article. Now that is it from this part, everything cannot be covered in any number of parts but at least it will help someone to get started in BASH scripting and its specifications for development. Have a Blast learning BASH. Happy Coding :) https://www.meetgor.com/posts/bash-game-numberjack Introduction OK! Learning BASH can be quite confusing without a proper goal in hand. So this will be a pretty good idea to start learning BASH and have a ton of Thu, 08 Jul 2021 00:00:00 UTC ## Introduction OK! Learning BASH can be quite confusing without a proper goal in hand. So this will be a pretty good idea to start learning BASH and have a ton of fun. In this little time, we'll make a Number game which I have designed myself last year in C++, which took about 3 months due to lazy research and wasting time. But I was surprised that I made this game within two hours in BASH. You can refer to the game instructions in this [repository at Github](https://github.com/Mr-Destructive/NumberJack). ## Concept The game will ask a number between 0 and 9 to the user. Then a list of 10 numbers shuffled in a random order will appear from of the user along with another list used for indexing the numbers from the array. The user has to select the index beneath its chosen number to proceed ahead. The game loops until the user has failed to enter the correct index of the number or the time for input was exceeded by 5 seconds. The user will get a point for every successful hit. So that is probably the introduction of the game so, let's dive into the specifications. ## Specifications of the Game in BASH The game is a number based which means it will need Arithmetic operators a lot. In fact, we'll need a few complex functions such as shuf. We will very frequently use while and for loops to perform some tasks such as filling and printing array and the game loop. We'll use some flag variables to indicate the current situation in the game and finally some arithmetic on arrays and numbers. ## Script Explanation The game is quite simple to understand. You just have to select the number beneath your chosen number within 5 seconds in the shell script. We will create a menu-like display in the terminal by simple echo command and formatting. Before the menu, we will have a while loop that will iterate until the user enters 3 which is stored in variable `ch` which is initialized to 0 in the beginning so as to enter the loop for the first time. A while loop starts with the do statement and ends at the done statement. ```bash while [ condition ]; do # statements done ``` For loop can be different based on the scenario. We'll use a range-based for loop to iterate over a range of numbers using the { } operators. For loop also has do as the beginning of the loop and done as the end of the loop. ```bash for i in {1..5}; do #statements done ``` We'll also use some If-else statements just to check for the correct user input and checking the exit status. The if statements have `then` to start the block and `fi` to end the if block. ```bash if [ condition ]; then #statements elif #statements else #statements fi ``` We use a read statement with the argument -p to have a prompt to the user for some information on the input. The input of choice from the menu i.e 1 to play, 2 for Instructions, and 3 to exit are stored in the variable `ch`. If the input is 1, the game will start and it will ask the user for the number `n`, which is the number used throughout the loop until the game is over. Now we have the number for the rest of the game, we need to generate the list for the user to select the number from. We will have a flag sort of to check if the user has entered the correct number which is `c`, this will store 0 for correct input(number x) and 1 for incorrect input. It is initialized with 0, again to enter the while loop once before the generation of numbers. To generate and **shuffle 10 numbers which should not have any repeated numbers**, as it can have multiple numbers which might be unfair also it might happen that the number chosen by the user might not be present due to repetition. So to avoid that mischief of pseudo-random numbers we have to generate distinct 10 numbers from 0 to 9 in this case. For that, we are gonna use a command in BASH called `shuf` which can create some permutation of the elements in a list/array or a sequence of numbers in an input stream. We are gonna use `shuf` to generate a random sequence of 10 numbers from 0 to 9 using the command `shuf -i 0-9 -n 10`.  You can see it generated a list of shuffled numbers between 0 to 9 so there are 10 numbers. We'll store this in result an array to access and print them later. You can refer to [this](https://www.geeksforgeeks.org/shuf-command-in-linux-with-examples/) and [these](https://www.howtoforge.com/linux-shuf-command/) articles for understanding shuf. The main thing is taken care of, now we need to print the list and also print another list to indicate the index of numbers to the user. We will print the list without a for loop using the `@` variable. If you are new to BASH and want a bit guide on BASH please do check out my series on [BASH scripting](https://techstructiveblog.hashnode.dev/series/bash-scripting), I have this all covered. So using `@` we can print the entire array in BASH.  To print the lower list of indices, we'll use a range-based for loop i.e it will iterate over a range(in this case between 1 to 10), and assign each element the value of the counter i.e from 1 to 10. We are using `r` as the shuffled list and `a` as the indices list. And print this array with the same method. After the generation and printing of lists are complete, we'll take the input from the user for the index of his/her number. We'll use an argument in read known as timeout, which will give a stop to the input stream after several seconds provided in the argument. In this case, we will use 5 seconds as a timeout for the input of the index. `read -t 5 -p "Enter the index of your number : " x ` We'll store the input in `x` variable and access it later for verification. Next, we will check if the input was done before the timeout or not. For this, if the user input before timeout, we can proceed ahead but if the time was over, then we'll get an exit status above 128 so we use this as a checker for the timeout in the input. I came to this via this [article](https://www.linux.org/threads/exit-script-by-timeout-if-delay-of-read-input-in-command-line.15905/), really very helpful. We will break the loop and make the flag `c` as 1 indicating an improper input and thus it'll show "GAME OVER". But if you were fast enough then we'll check that the index of the shuffled array has your chosen number or not, we used this `${r[$(($x))-1]} -eq $n` to check for the correct number. Why -1? If you remember indexing in the array by default starts with 0, as we have started the second list from 1 hence every element will become offset by 1 hence to avoid that we'll subtract one to refer to that index. If the index of the number was equal and correct, well done we'll increment the counter of points `p` by one and if it was incorrect, the flag will be set to one as previously said and we'll break the loop. After coming out of the loop, we'll check if the status flag `c` was 1 if yes, then print the GAME OVER and display the points earned. And that is it. Let's take a look at some gameplay :)  ## BASH Script ```bash #!/bin/bash echo -e " NumberJack " ch=0 while [ $ch -ne 3 ]; do echo " PLAY : Hit 1 and enter. HELP : Hit 2 and enter. EXIT : Hit 3 and enter. " read -p "Enter your choice : " ch if [ $ch -eq 1 ];then x=0 c=0 p=0 read -p "Enter any number between 0 and 9 : " n while [ $c -eq 0 ]; do x=11 r=($(shuf -i 0-9 -n 10)) echo "${r[@]} " for i in {1..10}; do a[$i]=$i done echo "${a[@]} " read -t 5 -p "Enter the index of your number : " x if [[ $? -gt 128 ]]; then c=1 break fi if [ ${r[$(($x))-1]} -eq $n ];then echo "Great" ((p=p+1)) else c=1 break fi done elif [ $ch -eq 2 ];then echo "HELP: INSTRUCTIONS TO PLAY THE GAME. " else break fi if [ $c -eq 1 ];then echo -e " GAME OVER " echo "You scored $p points" fi done ```  This is the final bare-bones script without any help instructions just keeping the script simple. I hope you learned something from the game development in BASH. This is just a fun little project and a cool way of learning certain concepts in BASH such as loops, conditional statements, and arithmetic. Have FUN. Happy CODING :) https://www.meetgor.com/posts/bash-guide-p2 Introduction In this part, topics such as switch cases, mathematical expression, arrays, and strings. This won't be an in-depth guide to understand each and eve Fri, 02 Jul 2021 00:00:00 UTC # Introduction In this part, topics such as switch cases, mathematical expression, arrays, and strings. This won't be an in-depth guide to understand each and every concept, but to make users aware of the things and features in Bash. This also would not be an absolute basic guide, I expect to have some basic programming knowledge such as binary systems, logical and mathematical concepts. Don't worry, you won't be bombarded with commands, I'll just explain with easy examples to get started. Topics to be covered in this part are as follows: - User Input - User Prompt - Changing the Delimiter - Password as Input - Limiting the length of Input - Cases - Arrays - Declaring and Printing Arrays - Number of elements in an array - Splicing the array - Inserting and Deleting elements - Strings - Making Substrings - String Concatenation - String Comparison - Arithmetic in Bash - Integer Arithmetic - Floating-Point Arithmetic # User Input Taking user input in Bash is quite straightforward and quite readable as well. We can make use of `read` command to take in input from the user. We just specify the variable in which we want to store the input.` read x` Here, the input will be stored in x. We can also pass in certain arguments to the read command such as -p (prompt with string), -r ( delimiter variation), -a(pass to the array), and others as well. Each of them will make the foundation of various complicated tasks to perform in logical operations. ### User prompt argument The -p argument will prompt the user with a string before they input anything. It makes quite informative and useful user input. This becomes quite a useful argument/parameter to make it quite readable and understand what to do directly with much hassle. The below is the general syntax of passing the argument to the read function. ```bash read -p "Enter the number " n ``` ```bash #!/bin/bash read -p "Enter the number " n echo "The inputted number was $n" ```  In this example, we can prompt the user with the string **Enter the number**, and it gives certain information to the user about what to input. ### Changing the delimiter Next, we can make use of -r which depending on the use case, we can change the delimiter while taking the input. ```bash #!/bin/bash IFS='/' read -p "Enter the file path : " user project app echo $user $project $app ```  In the above script, we separated the path of a directory(user-entered) into separate components such as the username, project name, and the app name, this can get pretty important and a great tool for automation of making project and application structures. At the beginning of the command, we use IFS which stands for Internal Field Separator, which does the separation of variables based on the field provided, in this case it was `//`, you can use any other field characters appropriate to your needs. This command will change the delimiter, by default it uses spaces or tabs etc to identify distinct input variables but we change it to other internal field separators such as / , \ ,- , \|, etc. This can make the user input more customizable and dynamic. ### Password Typing We can hide the input from the screen so as to keep it confidential and keep sensitive information such as passwords and keys private and protected. ```bash #!/bin/bash read -sp "Password: " pswd echo "the password was $pswd" ```  We used the -s to keep the input hidden, the screen doesn't reflect what the user is typing, and -p for the prompt to offer the user some information on the text. ### Limiting Length of Input We can limit the user to only a certain number of characters as input. It becomes quite useful in constrained environments such as usernames and passwords to be restricted with a certain limit. ```bash #!/bin/bash read -n 6 -p "Enter the name: " n echo $n ```  In the above script, we demonstrate the limit of characters of 6 in the variable n. This restricts the user with only the first 6 characters, it just doesn't exceed ahead, directly to the next command. ### Passing to the array Another important argument to be passed after read command is -a which inserts the value to the array elements. ```bash #!/bin/bash read -a nums -p "Enter the elements : " for i in ${nums[*]}; do echo -e "$i " done ``` In the above script, we have used array, don't worry, I'll explain it in the coming sections of this part. We have assigned an empty array and made the user enter those arrays, they are space-separated values. We have used the -a operator to insert the input to the elements of the array. The for loop is range-based which means it will do certain commands until there are no elements left in nums. The command `${nums[@]}` indicates every element in the array nums. # Cases Cases are quite a good way of replacing nested if-else statements to make them nice and readable in the script. Cases in Bash are quite powerful and easy to use compared with C/ C++ styled switch cases. The general structure of using a case in Bash is as follows: ```bash case variable in pattern 1) statements ;; pattern 2) statements ;; pattern 3) statements ;; pattern 4) statements ;; *) statements ;; esac ``` It follows a particular pattern if it matches it stops the search and executes the statements, finally comes out of the block. If it doesn't find any match it redirects to a default condition if any. ```bash #!/bin/bash read -p "Enter a name : " n case $n in admin) echo "You are logged as root" ;; unknown) echo "A hacker probably" ;; manager) echo "Weolcome Manager!" ;; *) echo "A normal person" ;; esac ```  Seeing the above example, it is quite clear, that it looks quite structured and readable than the nested ladder of If-else statements. Cases are rendered based on the variable after the `case` keyword. We use the patterns before `)` as making the match in the variable provided before. Once the interpreter finds a match it returns to `esac` command which is `case` spelled in reverse just like `fi` for `if` and `done` for `do` in loops :) If it doesn't match any pattern, we have a default case represented by `*)` and it executes for any non-matching expression. ## Arrays Arrays or a way to store a list of numbers is implemented in Bash identical to most of the general programming languages. ### Declaring and Printing Arrays We declare an array similar to a variable but we mention the index of the element in the array(0 based index). We can also simply declare an empty array using the declare command `declare -A nums` ```bash #!/bin/bash nums[0]=7 nums[1]=5 nums[2]=8 for i in ${nums[@]} do echo -e "$i " done ```  The above script initializes an array with some hard-coded elements, surely you can input those from the user. For printing and accessing those elements in the array, We can use a loop, here we have used a range-based for loop. You can use any other loop you prefer. The iterator is " i " and we use $ to access the values from the array, we use `{}` as we have nested expression for indexing the element and `*` for every element in the array ( `@` will also work fine ), that's why range-based for loops make it quite simple to use. And we have simply printed " i " as it holds a particular element based on the iteration. OR ```bash #!/bin/bash declare -A nums=( [0]=44 [1]=45 [2]=46 ) echo "${nums[@]}" ```  The above script uses declare an array, it can be empty as well after the name declaration. We used the `()` to include the values in the array, using indices in the array we assigned the values to the particular index. If you just want to print the elements, we can use `${nums[@]}` or `${nums[*]}`, this will print every element without using any iteration loops. ```bash #!/bin/bash nums[0]=7 nums[1]=5 nums[2]=8 echo "${nums[@]}" ```  ### Number of Elements in the array To get the length of the array, we can use # in the expression `${nums[@]}`, like `${#nums[@]}` to get the number of elements from the array.  Since we had 4 elements in the array, it accurately printed 4. ### Inserting and Deleting elements from Array We can push elements to the array using the assignment operator. `nums=(${nums[@]} 76) ` This will push 76 into the array, i.e. in the last index( length -1 index). ```bash #!/bin/bash nums[0]=7 nums[1]=5 nums[2]=8 nums[3]=19 nums=(${nums[@]} 76) echo "${nums[@]}" echo "Length of nums = ${#nums[@]}" ```  As you can see the element was added at the end of the array, you can also specify the index you want to insert. We can use `unset nums[3] ` to delete the element at the particular location or we can pop back (delete from the end) an element from the array using the index `-1` from the array using the following command. ```bash unset nums[-1] ``` Any index provided will delete the element at that location by using unset. By using -1, we intend to refer to the last element. This can be quite handy and important as well in certain cases. ```bash #!/bin/bash nums[0]=7 nums[1]=5 nums[2]=8 nums[3]=19 unset nums[-1] echo "${nums[@]}" echo "Length of nums = ${#nums[@]}" ```  There you can see we removed the element using unset. ### Splicing an Array We can splice the array to print/ copy a portion of the array to another one. ```bash echo "${nums[@]:1:3}" ``` Using two colons and numbers in between them, we can print in this case certain elements in the array from a particular range. Here the first number after the colon is the starting index to print from(inclusive) and the next number after the second colon is the length to which we would like to print the element, it is not the index but the number of elements after the start index to be spliced ```bash #!/bin/bash nums[0]=7 nums[1]=5 nums[2]=8 nums[3]=19 nums[4]=76 newarr=${nums[@]:1:3} echo "${newarr[@]}" echo "${nums[@]}" ```  In this case, we have copied the slice of an array to another new array using the double colon operator. We added the elements from the 1st index till `1+3` indices i.e till 4. 3 is not the index but the length till you would like to copy or print. This was a basic introduction to arrays, definitely, there will be much more stuff I didn't cover. Just to give an overview of how an array looks like in BASH scripting. Next, we move on to strings. # Strings Strings are quite important as it forms the core of any script to deal with filenames, user information, etc all contain strings or array of characters. Let's take a closer look at how strings are declared, handled, and manipulated in Bash scripting. ```bash s="World" echo "$s" ```  Strings are again declared as normal variables but are enclosed in double quotation marks. And we access them in the exact same way as we do with variables. If you were to use single quotes instead of double quotes Bash would not interpret the variable name as a variable, it would print the name literally and not the value of the variable, So prefer using double quotes in echo and other commands to make variables accessible. ### Making Substrings We can even splice the string as we did with the arrays, in strings we can call it substrings. The syntax is almost identical as we just have to get the variable name. ```bash #!/bin/bash s="Hello World" p=${s:6} echo $p q=${s::5} echo $q t=${s:4:3} echo $t ``` In the above script, we had a base string 's' which was then sliced from the 6th index to the end, If we do not pass the second number and colon, it interprets as the end of the string and if we do not pass the first number, it will interpret as the first character in the string. We sliced s from the 6th index till the end of the string and copied it in the string 'p''. In the 'q' string, we sliced the first 5 characters from the string 's'. In the 't' string we sliced starting from the 4th index and 3 characters in length i.e till 7th index, not the 7th index. We can use the # before the variable name to get the length of the string as we saw in the array section. So we can use the `echo ${#s}` command to print the length of the string where s is the string variable name. ### String Concatenation String concatenation on Bash is quite straightforward as it is just the matter of adding strings in a very simple way. ```bash #!/bin/bash s="Hello" p="World" q="$s $p" echo $q ```  The space between the two variables is quite literal, anything you place between this space or the double quotes will get stored in the variable or get printed. ### String Comparison Moving on to the string comparison in Bash. String comparison is quite complex in certain programming languages but it's quite straightforward in some languages such as Bash. We can compare strings quite easily in Bash, either they are equal or they are not, it's just comparison operators to perform the heavy-lifting for us. ```bash #!/bin/bash s="hello" p="Hello" if [ $s = $p ]; then echo "Equal" else echo "Not equal" fi ```   From the above code, it is quite clear that the strings as not equal and we compared them with the "equality" operator (=) and checked if that condition was true, and perform commands accordingly. We can also check if the strings are not equal using `!=` operator and we can perform commands based on the desired logic. We also have operators to compare the length of the strings. We can use `\<` operator to check if the first string is less than the second string(compared characters in ASCII). And check if the first string is greater than the second string using `\>` operator. ```bash #!/bin/bash s="hello" p="Hello" if [ $s \< $p ]; then echo "$s is Less than $p" elif [ $s \> $p ]; then echo "$s is greater than $p" else echo "Equal" fi ```   Here, we are using the ASCII equivalent of strings to compare them as it gives an idea in terms of the value of the strings. We see that 'h'( 104)has a greater ASCII value than 'H" (72) which is why we see the shown outcome. We also have operators to check for the string being empty or not using the -z operator. Also, we have arguments to pass to the string comparison to check for non-empty strings as well, specifically for input validation and some error handling. We can quite easily use -n to check for non-empty string and -z for the length of the string being zero. ```bash #!/bin/bash read -p "Enter a string : " s if [ -s $s ]; then echo "Empty Input" else echo "Valid input" fi ```  So the string topic is quite straightforward and self-explanatory as it doesn't how that much complexity but is still powerful and convenient to use. # Arithmetic in Bash Performing any Arithmetic operations is the core for scripting. Without arithmetic, it feels incomplete to programmatically create something, it would be quite menial to write commands by hand without having the ability to perform arithmetic operations. ### Integer Arithmetic Firstly we quite commonly use operations on variables, so let us see how to perform an arithmetic operation on variables in Bash. We use double curly braces to evaluate certain results of the operations performed on variables. ```bash #!/bin/bash x=4 y=9 z=$(($x * $y)) echo $z ```  We use double curly braces in order to evaluate the operations performed on the variables inside them. We definitely have to use the $ symbol to extract the value of the variable. We can surely use operators such as addition(`+`), subtraction(`-`), multiplication(`*`), division(`/`), and modulus(`%`, it stores the remainder of the division,17%3 gets you 2) in statements. We can also perform operations such as `<<` to do left bitwise shift and `>>` right bitwise shift to shift the binary digits in left tor right respectively in a variable. There are also logical operations such as Bitwise and logical AND(`&`), OR(`|`), EX-OR(`^`), and ternary expressions. Alternative to double curly braces is `expr`, expr allows you to freely wherever you need to evaluate an arithmetic operation. Though this is not native in-built in shells, it uses a binary process to evaluate the arithmetic operations. It can also defer depending on the implementation of such commands in various environments. We can also use the `let` command to initialize a variable and perform expressions in the initialization itself. ```bash #!/bin/bash let a=4 let b=a*a let c="b/(a*2)" echo $b ``` We can perform quite complex operations using simple implementation using `let`, this allows much readable and bug-free scripts. If you would like to perform operations with brackets and other operations you can enclose the expression in double quotation marks. ### Floating-Point Arithmetic Performing floating-point arithmetic in Bash is not quite well though. We won't get 100% accurate answers in the expressions this is because it is **not designed** for such things. Doing **things related to floating-point is a bad idea**, Still, you can improve the precision to a little extent to do some basic things. I **don't recommend this** only do this if there are no other options. ```bash printf %.9f "$((10/3)) ``` The result of this is 3.0000000..064 roughly, which is pretty bad. Bash at its core doesn't support floating-point calculations. But there is good news, we have [awk](https://en.wikipedia.org/wiki/AWK) and other tools such as [bc](https://en.wikipedia.org/wiki/Bc_(programming_language)) and others which is planned for the next part in the series. I'll explain awk just for floating-point here, in the next part, I'll cover it in depth. ```bash #!/bin/bash a=10 b=$(echo | awk -v a="$a" '{print a/3}') echo $b ```  WOW! That is to the point, but that was a lot of hassle using echo but printing nothing! HH? OK, you see certain things can get really annoying when things aren't supported natively. Firstly, we use | to pipe echo command with awk, the echo command doesn't do anything just a way to use awk command in assigning variables here. Then the general syntax of the awk command is ` awk -options -commands`. In this case, we are using -v as an argument and passing in an as a variable which is equal to a, which is stupid and silly but that is what it is, you can name any variable name you want. Then we simply have to use the variable in the print function which generally evaluates the expressions or other operations and returns to the interpreter. And that is how we print the expression, Phew! That took a while to do some silly things, But hey! That's possible though. That is the basic overview of Arithmetic in Bash, you can also perform logical operations in it which is very easy and can be understood on a quick run-through in the [documentation](https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Arithmetic-Expansion). I hope you understood the mentioned topics and what are their use cases depending on the requirements. Some topics such as positional parameters, tools and utilities, dictionaries, and some other important aspects of Bash scripting will be covered in the next part. Happy Coding. https://www.meetgor.com/posts/vim-plus-teminal Vim and Terminal!? Vim was made to work with the command line. Many beginners do not understand what are the true capabilities of Vim, myself included:) Vim can Tue, 29 Jun 2021 00:00:00 UTC ## Vim and Terminal!? Vim was made to work with the command line. Many beginners do not understand what are the true capabilities of Vim, myself included:) Vim can run terminal commands without leaving the text editor, open an instance of a terminal, work with shell environments, and other things depending on the use case. ## Running Terminal/ shell commands from within Vim You can run the commands from inside of Vim by just using `:!` before the command, this means you have to be in command mode. Just after being in command mode, the ! or bang operator will execute the command typed after it from the terminal(Linux/ macOS) or your default shell(Windows -> CMD/Powershell). ``` :!pwd ``` The above command from vim will redirect to the terminal and show the output of the command and return on pressing any key. In this case, it will execute the PWD command and just wait for the user to enter any key to return to Vim. The following is an example of how it could be used from Vim in Windows using Powershell as the default shell.  In Windows, dir is equivalent of ls for Linux. That was not the best example of how a terminal can be used at its best, You can also use a logical operator from within vim to run multiple commands at once. ### Running programs/ code from Vim on terminal This becomes quite a great feature for making Vim from a text editor to an IDE, this can be paired with Keymaps i.e when the user types certain keys, the command gets executed making the code run from the terminal. I have already used this feature to set up Vim for python, bash, and other programming languages. Also, I have written an article about [keymapping](https://dev.to/mrdestructive/vim-keymapping-guide-3olb) and Vim setup for [Python](https://dev.to/mrdestructive/setting-up-vim-for-python-ej) and [Bash](https://techstructiveblog.hashnode.dev/vim-setup-for-bash-scripting), this will give you an idea of how to setup vim for any programming language. Vim can really shine in this kind of feature as it just becomes flawless and a smooth experience even for a beginner. We just have to run the compile the code and run its executable/ output file, rather for python and other interpreted languages, we have to just pass the file name to the interpreter and that's it. ## Opening instance of Terminal within Vim. Vim can also create an instance of the terminal within its window by making a split. This is quite similar to VS Code and other Text editors that have the functionality to create an instance of the terminal within itself. This feature is useful for developing complex systems and depending on the use case, it can be quite important and efficient as well. The terminal can be created in various ways the most preferred way is by typing in `:term` from Vim. This will create a horizontal split from the current editor and split it into half. You can change the size of the split using the mouse according to your preference.  Here Vim has certain variables and shortcuts to make things even simpler, say you want to parse the current file to the terminal for execution. You can surely type the name manually or you can be a bit smarter and use % instead, the `%` symbol will parse the file name along with the extension in the terminal. Also `%:r` will parse filename without the extensions(.txt/.py/etc) to the terminal. There are many things you can do with terminals surely, but with Vim that even goes further than the limits. Terminal/command line is quite important in any development environment as it is an interface for the user to interact with the Operating System. Vim is quite powerful and behaves as a gecko for programmers because it changes itself according to our needs flawlessly and **efficiently**.  Integrating Terminal into a Text Editor truly lights up the environment for development. It becomes an easy and enjoyable experience to test out the code without wasting much time on the actual execution process. Surely it needs time to set up the environment to speed things, for that understanding of the programming and development environment is required. Happy Viming :) https://www.meetgor.com/posts/vim-movement-speed Introduction OK! Vim and movement are like bread and butter or failed brakes. To become a proficient Vim user, you need to move in Vim very effectively. You don Sat, 26 Jun 2021 00:00:00 UTC  ## Introduction OK! Vim and movement are like bread and butter or failed brakes. To become a proficient Vim user, you need to move in Vim very effectively. You don't have to think about doing certain things, your fingertips should automatically move without wasting time thinking about it. I am sure, it takes time and effort but OH! it is so rewarding in the end. ### Why H J K L? First things first, unmap arrow keys and make a habit of using h,j,k, and l. Yes, this would not make any sense in the initial stage but that will make no sense for not using it later. The thing with H J K L is that they are quite flexible to work with and if you use it with numbers you would navigate much faster than arrow keys. Such as `10j` will move you 10 lines down at a time in normal mode. These keys are used with many of the other key shortcuts and commands which just make it most important to begin learning to move around in Vim with H J K L. ## Moving Horizontally. This is quite the common movement that every programmer uses most of the time. This is also a much-neglected part when movement in Vim is concerned. To the basics, we use the following commands: `w` -> **move forward by a word (considering punctuations as separate words).** `W` -> **move forward by a word (punctuations ignored).** `b` -> **move backward by a word (considering punctuations as separate words).** `B` -> **move backward by a word (punctuations ignored).** `e` -> **move to end of a word (considering punctuations as separate words).** `E` -> **move to end of a word (punctuations ignored).** `0` -> **move to the beginning of a sentence.** `$` - **move to the end of a sentence.** Those are the most useful and common commands for moving across the line. Don't forget to use the number before the command to repeat the task for that number of times. Like for example, if you would like to go 6 words ahead type in `6w`. This can improve your thinking and typing as well, good signs of a programmer Eh! ## Moving Vertically. To move vertically we can imagine moving within a file or the block of code. For moving in a file, the following are some useful commands. `gg` -> **move to the beginning of a file.** `G` -> **move to the end of a file.** `Ctrl + e` -> **move the screen down without moving the cursor.** `Ctrl + y` -> **move the screen up without moving the cursor.** `Ctrl + f` -> **move forward one entire screen.** `Ctrl + b` -> **move backward one entire screen.** `Ctrl + d` -> **move forward half screen.** `Ctrl + u` -> **move backward half screen.** This just was moving around the screen and now a bit programmatic movement. We will see some keystrokes to move in code blocks or code snippets very efficiently. `gd` -> **move to the local declaration of any code.** `gD` -> **move to the global declaration of any code.** `%` -> **move between pairs of ( ), { }, [ ] or any other type of such braces.** `{` -> **move to the next paragraph/ code block/ function/ etc)** `}` -> **move to the previous paragraph/ code block/ functions/ etc)** `fa` -> **move to the next occurrence of the character 'a' in a sentence.** `Fa` -> **move to the previous occurrence of the character 'a' in a sentence.** `ta` -> **jump to before of the next occurrence of the character 'a' in a sentence.** `Ta` -> **jump to after of the previous occurrence of the character 'a' in a sentence.** The above might be quite handy key shortcuts in moving in a large code file. Even in files with complex variable names and structures, this can be quite handy. ## Search and navigation. Searching is quite a time-consuming task, especially when the code is quite complex and has a lot of variables and all. Vim shines in many of such aspects where people think it's dead. It rises from the ashes to produce a performance-driven experience like any other modern IDEs though it requires a bit of research:) Here are some commands that will make searching and navigating around it quite a lot easier. `*` -> **next occurrence of the word under the cursor.** `#` -> **previous occurrence of the word under the cursor.** `n` -> **next occurrence of the word searched pattern.** `N` -> **previous occurrence of the word searched pattern.** The above commands will also work if you search the pattern from the command mode. `/pattern` or `?pattern` Enter and navigate to the next(`*` or `n`) and previous(`#` or `N`) occurrence of that pattern match. ## Moving across files. Moving across files without any plugins or file explorer is often considered tricky or impossible for some people but there is a way. You can switch between files using the following commands: `Ctrl + O` -> **move in the previously opened file.** and `Ctrl + I` -> **move in the next file.** We also can use `Ctrl + ^ ` to move the previous two opened files. If you want to switch from buffers, you can use `:bn` for moving into the next buffer, and `:bp` to move in the previous buffer. You always have an option to move from a buffer using the file name `:b filename` or using the index as `:bindex`. ## Moving between Tabs. People rarely use Tabs as far as I have seen, but they are quite useful and provide the polish off just as robust IDEs. `:tabnew filename` -> **create a Tab of a file.** `gt` -> **move to the next tab.** `ngt` -> **move to the nth tab.** `gT` -> **move to the previous tab** `:tabo` -> **close all the tabs except the current one.** `:tabc` -> **close the tab.** `:tabm n` -> **move the current tab to nth position.** ## Movement in Marks. Marks are used for some quite large files and code-bases. It is used to move from one mark(kind of a bookmark) to another using few key commands, marks are generally created when you would go to a particular code block or any part of the file again and again. Some of the quick navigation using maps are the following. `mn` -> **set the current position as mark 'n'.** `<backtick>n` -> **jump to the position of mark 'n'.** ` `0` -> **jump to the position where vim was last exited.** ` `"` -> **jump to the position when the last edit was made in the file.** ## Split Windows Movement `Ctrl + w + r` -> **move the split down.** `Ctrl + w + R` -> **move the split up.** `Ctrl + w + h` -> **jump to the left split.** `Ctrl + w + j` -> **jump to the split down.** `Ctrl + w + k` -> **jump to the upper split.** `Ctrl + w + l` -> **jump to the left split.** You can use Caps H J K L to move the leftmost, bottom, uppermost, rightmost split respectively. That just was quick to make you enough faster than previous hassles. That was probably it, these were some tricks and shortcuts to move around Vim pretty effectively and smoothly. Moving around Vim can be quite complicated at once, but it's just finding the key shortcuts to make you feel and glide in VIm. There might be quite a lot of shortcuts missing, If you have any quicker shortcuts, Please let me know in the comments. Happy Viming :) https://www.meetgor.com/posts/bash-guide-p1 Introduction If you are new to BASH and Linux, don't you worry the community is the driving force here. If someone's stuck somewhere, the platforms, forums, and Tue, 22 Jun 2021 00:00:00 UTC ## Introduction If you are new to BASH and Linux, don't you worry the community is the driving force here. If someone's stuck somewhere, the platforms, forums, and blogs are ready to help anyone there. BASH is a short term of Bourne-Again Shell, it is a shell interface that has much more capabilities and functions than the simple Bourne shell(sh). It has some quite remarkable features and it is even capable of calling itself a programming language in some sense. Without wasting any time on the introduction, let's keep the article rolling. In this part, I'll try to cover the basics of the following topics: - **Structure of Bash Script.** - **Variables.** - **If-else Conditional Statements.** - **Loops.** 1. For loop 2. While loop 3. Until loop - **Functions.** Firstly, you should have some basic understanding of Linux commands such as listing directories, creating, making editing files, and some tiny little tasks. Bash scripting is the way to do these in a programmatic way, that's why it is called scripting. ## Understanding a Simple Shell script ```bash #!/bin/bash echo "Hello World" ``` That is such a simple elegant script, isn't it? Well, the first command is called the she-bang which tells the Shell to execute or run the file as a Bash script or from a Bash interpreter. The next command is a simple echo which is used for printing text to the screen/console. She-bang is the path to the Bash interpreter. So, it basically redirects the shell to execute the file/script in a Bash environment.  To execute the script we have many ways, either use Bash, source, or execute it as a shell-script by making it executable from the path. In this case, I used Bash to run the script, we'll see others as well. The core structure of the Bash script is quite simple, we can make the format of the script according to the paradigm used and objective of the script. For basic scripts which has utility functions we normally declare those in the beginning after the she-bang header. After the function, we can have the core main part of the script. It was enough and important here to understand the purpose of the she-bang header and how to execute a shell script. ## Variables Definitely, we need variables to store some value which we are gonna use again and again. Bash has variables but without any specific data types, you can store anything in anything. It becomes a mess in comparing to variables and their values :( Though it might be a feature, not a bug sometimes. Let's create some variables in a Bash script. ```bash #!/bin/bash x=2 echo $x ``` To create a variable, we simply write the variable name and assign it to whatever we want. **DO NOT leave spaces before and after assignment operator(=), it won't work**. To access the value of the variable we use the $ symbol. That is about the variables, If you want to perform some arithmetic on variables, it is covered in the further sections.  From the above script, we outputted the value of x to the console. We also executed the script file using the source command. ## If-else Conditional statements If-else conditional statements are quite the fundamentals of any logical operations performed in the programming world, so Bash has them implemented pretty much the same way as other shells do. ```bash #!/bin/bash a=9 if [ $a -lt 10 ];then echo "Less than 10" fi ```  We use "If", followed by "then" to execute the mentioned commands after that until it hits "fi" or else statement, we'll see else and if-else block after this. That is a basic If statement, here if we are comparing numbers we use -lt for less than, -gt for greater than, -eq for equals to, -ne for not equals to, -le for less than equals to, and -ge for greater than equals to. For string comparison, we use symbols such as < for less than, > for greater than, = for equals to, != for not equals to. ```bash #!/bin/bash a="ERROR" if [ $a = "ERROR" ]; then echo "FAILURE" else echo "SUCCESS" fi ```  In the above example, we have used the if-else block, comparing a string with other and using the = operator to compare. It's quite interesting that Bash has string comparison built-in, unlike C/C++ where we have to depend on external libraries. We have used chmod to make the script file executable to anyone using the system. And then we simply put in the path to the file to run it. ```bash #!/bin/bash c=3 if [ $c -eq 0 ];then echo "Success" elif [ $c -eq 1 ];then echo "Failure" else echo "Invalid Input" fi ```  From the above script, we used if-elif-else statements to evaluate different conditions. We had use -eq to equate the value of the variable to the number we want to compare with. That was self-explanatory logic. ## Loops We have 3 types of loop statements in Bash, they are: 1. For loop 2. While loop 3. Until loop ### For loops In for loop, we have the freedom to use in range-bound or C-like for loops. Let us take a look at both of them using examples. ```bash #!/bin/bash for i in {1..5} do echo $i done ```  The above loop was used as a range-based loop, which loops through 1 and 5 inclusive. We use {} to use it as the range. As "then" and "fi" in if conditions, we have "do" and "done" in loops. Between the do and done statements, we can type in the statements we want to loop. ```bash #!/bin/bash read -p "Enter the number : " n for (( i=0;i<$n;i++ )); do echo $i done ```  The above for loop is a typical C-style for loop which takes 3 arguments, the initializing iterator, the condition, and the incrementor. We surround the arguments with double braces followed by a semi-colon. The rest of the syntax is identical to the previous for loop style. ### While loops While loops are used quite commonly in Bash and the syntax is quite straightforward. ```bash #!/bin/bash read -p "Enter the number : " n while [ $n -gt 0 ] do echo $n i=$((--n)) done ```  The above while loop has a condition for the number to be greater than zero. We take in the input from the user using the command read and store it in the variable n, the -p is an argument to prompt the user with text before the input. We use the decrement operator to decrement the iterator. The syntax is quite similar and easy to understand. As usual in loops, we have do and done to start and end the loop. ### Until loops. Until loop is a while loop but with the opposite condition, we loop until a certain criterion is not matched. ```bash #!/bin/bash read -p "Enter the number : " n until [ $n -eq 0 ] do echo $n n=$((--n)) done ```  In the loop, we iterate over and over again until n becomes 0. Until is simply to exit from the loop until a certain condition is met. The rest of the syntax is again the same as the other loop with do and done statements. ## Functions Functions are the part and parcel of any script, we don't use it necessarily, but they come in handy quite some times and serve the purpose absolutely well. We can customize what we want to return from the function depending on the needs. ```bash #!/bin/bash function add(){ echo $(($1+$2)) } add 3 6 ```  In Bash functions, we do not pass arguments inside brackets, we have to pass in parameters as space-separated values after the function name in the function call. This also means that we can pass any number of parameters to a function, but only we should handle them properly. Otherwise, there is no use in passing unlimited parameters to a function. This is really powerful but it needs to be used wisely to have its full potential. Also, it not mandatory to use the keyword "function" before the name, So you could also write just the name and the rest of it as it is. ```bash #!/bin/bash add(){ echo $(($1+$2)) } add 3 6 ``` The return statement in functions is also quite an important aspect in logical programs. But it is not like returning values from a function, it is like returning the exit status of a function scope, so it can return inappropriate behavior from the shell depending on the return value. ```bash #!/bin/bash add(){ return $(($1+$2)) } add 6 13 echo $? ```  Here we return the addition of two numbers and we use the internal variable ? To access the exit status of the function. The ? Is an internal variable in Bash, which holds the exit status of the last executed command. In this case, it was the function call and its return statement was stored in it. As said, it can become quite buggy to exit the function scope with wired return statements, so to avoid those we can make use of global variables. ```bash #!/bin/bash add(){ n=$(($1+$2)) } add 5 16 echo $n ```  In the above program, we use n as a global variable to access out of the function scope. Function in Bash can also return multiple values at once but that can be buggy at times, so I don't recommend that. So, that is the basics of Bash functions covered. This is it from the Bash scripting guide Part-1, I'll cover more topics in the upcoming parts of the series. https://www.meetgor.com/posts/bash-intro Introduction BASH or Bourne Again Shell is an interface(program) between the user and the Operating System. BASH is just an application just like others, but no Fri, 18 Jun 2021 00:00:00 UTC ## Introduction BASH or Bourne Again Shell is an interface(program) between the user and the Operating System. BASH is just an application just like others, but not an ordinary one. It is quite powerful when used to its potential. It is used in various aspects of computing. No, it's not only for server admins, but it specializes in that field though. We'll see how we can use Bash to perform several tasks quite powerfully. Not just automation, you'll be surprised what Bash is capable of. ## BASH - The program that powers the world. BASH is just like any other shell such as z-shell, c-shell, Korn shell, and Bourne shell. BASH is a variant or update version of the BOURNE shell. Technically it has quite advanced features than the Bourne shell, such as arithmetic operators, signal handling, command-line completion, and tons of quality of life improvements. So what can Bash do? Well pretty much anything, it is close enough to call itself a programming language but not quite enough. You can call it the programming language of the command line. Bash also has built-in features such as variables, loops, if-else conditional statements, functions, and all but there are no concrete data structures to work with. Some of them quite useful structures are built-in such as arrays and dictionaries. It can perform arithmetic, logical and programmatical operations inside of a terminal or command line. It can make use of some other programs such as grep, awk, sed, cURL, find, wc, the list is quite large. Most of the commands are quite versatile and can be integrated with each other quite well. Certain concepts such as command piping, brace expansion, and others provide a great integration and feasibility to make quite a lot of applications using Bash. Certain commands such as grep, cURL, sed, cat, touch are quite incredible and are used a lot in day-to-day use cases. Let's take a quick look at how actually we can use Bash to perform a basic task.  From the above gif, I edited a file without leaving the terminal. That can be lame but powerful in certain conditions. Anyways, that was just an example, but you can see where it can go. It can be extremely powerful for some repetitive tasks in programming or any other tasks. It can be also quite vividly used in the upcoming fields. ## Applications Well, Bash has some epic applications, some of them might surprise you but they are indeed liable to call themselves features of Bash. Let's dive in then. ### Automation "of course!" This is the most known feature of Bash as it stands upright on top of other tools and programs. Bash is most commonly used by system admins to automate repetitive tasks in managing a system. These tasks may include backing up files, managing file structures, monitoring the performance of a system, and debugging any errors and issues in the system on a regular basis to ensure the smooth functioning of a system. So, it makes quite a lot of sense to automate certain tasks that might waste time and effort to do them over and over again. Hence, Bash is the system admin's best friend and also for programmers to automate certain tasks and compilation of programs. ### File handling This is another aspect where Bash just makes things flawless, just flawless. It's much more easy, straight forward and intuitive to manage or handle files in Linux/ Unix operating systems. The tools and command-line utilities such as grep, awk, sed for matching patterns using regular expressions, cat, tac, touch to create and make changes to the file. There are a plethora of ways to make files and directories using Bash utility tools. We can also make a bunch of files at once, make the structure of a file system at once just using Bash and its commands. ### Programming environment This is a concept which many people use but are unaware of. People using Linux for programming are mostly using bash as their environment to make, compile, run files of the program. That is quite a powerful environment to work with as it is quite customizable and efficient as compared to Windows at least. Even macOS used Bash till a certain time but migrated to zsh for silly reasons about security issues on Bash. But still, Bash is the tool that most programmers use while interacting with the web servers or any other programs that deal with some kind of interface to the operating system especially Linux/ Unix. So, it is worth noting that Bash is the default development environment for many programmers. ### Networking Networking is another aspect where Bash also some quite remarkable tools to play around with. Bash has certain powerful tools like Nmap, aircrack-ng, ping, and many other networking tools which makes it quite a good choice for ethical hacking and network security as well. This domain is quite complex in terms of concepts as it makes the core IT domain. Bash also provides ways to manage the network using some well-equipped tools and programs to make it quite user-friendly. ### Web-Scraping "wait what!?" Yes, This is possible by making use of certain tools such as cURL to extract the content from the web and then using grep, sed, awk, or any other pattern/ regex matching tools to filter out the content. This might be very useful and informative for beginners trying to understand how web scraping actually works. Web-scraping can be learned this way so as to understand "Under the Hood" working of some utility and frameworks for web-scraping. This is some super great stuff for a simple shell to do in a standalone way though it may require a good skill of regexes and how the web works to make full utilization of the powers of Bash. ### Maybe More OK, now I just know about these many applications about Bash but if you know any other please let me know in the comments. Bash is a great tool and I don't why it's not quite famous but still used a lot. Maybe I am wrong here, just an opinion though. So, I hope you might have got some insights to dig deeper into Bash and learn to improve efficiency and skills. So, that's it from Bash and its application in a high-level view, there is a huge field and community for Bash, just because of some threats Bash is not going anywhere, still, zsh is getting replaced in most cases, but the concept and tools remain the same. At the end of the day, its the skill rather than the version of any tool that matters Happy Coding and have a blast learning Bash :) https://www.meetgor.com/posts/vim-keymaps Introduction Keymapping or Key binding is a process in which a user or a system can assign certain keys or commands to perform a particular task or commands. Th Mon, 14 Jun 2021 00:00:00 UTC ## Introduction Keymapping or Key binding is a process in which a user or a system can assign certain keys or commands to perform a particular task or commands. This can be quite useful in Vim as well as in other text editors to avoid some repetitive tasks and save time. In Vim this can be a great power to have for programmers as it can be really very flexible to set up for any programming language and it's environment. In Vim, you can map a particular keystroke combination to trigger a command or any operation. You can also map a key in place of a weird key combination as well. You can map a key to integrate several commands and make it run at a single key or without much of a hassle. Key Mapping is quite powerful in Vim, only your creativity and imagination are the limits here. Vim can really shine in such aspects where the user has the freedom to tailor the editor and his development environment as per his/her needs. ## Understanding the structure of Key binding You can basically map a key depending on the current mode you are in. So that means you can have the same key mapped to different commands depending on the mode. That is really flexible and powerful. Vim allows you to basically map in almost every mode such as normal, insert, visual, command, and any other existing modes. Let us see what a basic key mapping looks like. ``` {mode}{attribute}map {key} {command} ``` The key mapping is broken down by adding the mode to map the key, an optional attribute to change/modify the way the mapping behaves and we finally have the key and the command pair. If you want to test a map you can temporarily map in the command mode using the syntax for the map command or if you want a permanent mapping, you can add the mappings in your vimrc file. For a basic example let us map c to paste from the clipboard. ```vim :map c "+p<CR> ``` The above command maps the "c" key in the **normal/visual mode** by default to paste the contents from the clipboard. It is just an example of how a basic key mapping or binding can be used to save your time and efforts. This is a very small example to demonstrate the concept of mapping in Vim but it scales quite fantastically and smoothly. Now let's see how to view the mapping inside of the current vim buffer. We can simply type map, imap, or nmap in the command mode to view the mappings in the respective modes. This will display the maps which are currently present inside of the vim buffer, some might be temporary and most of them will be permanent. ``` :map ``` If you type in map being in the command mode, it will list out the maps. If you want to stop seeing the list just press q, and you will be back in the editor. You can also view the maps in the normal, visual, insert, or any other modes by prefixing n, v, i, or other modes with map. ## Key Mapping in Modes To map a key binding explicitly in the normal mode or any other modes, we have to prefix the word such as n for normal, v for visual or i for insert, and so on. The map will only work in the provided mode so the same keys may have several different maps depending on the current mode, but that can create a bit of confusion so avoid doing that. Let's map a key in the normal mode for example, ```vim :nmap <C-s> :w<CR> ``` This is pretty bad for Vim beginners as it will spoil the real Vim experience of saving with :w, If you feel comfortable using Ctrl + s to save a file, the above map will be perfect. It will map the key Ctrl + s by pressing :w and then pressing enter to save changes to the file. We can also map certain non-alphabetical keys such as CTRL, ALT, and others, it will help you in saving time to leave a mode and enter the mode again to edit the text. The following is a perfect example of such. ```vim :imap <C-c> <ESC>"+pa ``` The above command is **mapped in insert mode** as the mode is set to insert at the command's beginning. The CTRL+c keys are mapped to first Escaping out of insert mode and then pasting from the "+ register which is the device clipboard here. In the end, we again get back to insert mode just to the right of the current cursor position. So, that is how we can map the keys to do the task and save a bit of time. ## Adding Keymaps in vimrc file So you can now map in any available modes in Vim and test it out in te=he current buffer, but if you notice when you close the buffer, your custom keymaps will not be functioning as they remain in the buffer until you close it. To make it permanent you need to type the exact map command to the vimrc file. Just don't add ":" as it is not a command mode, it should be only the pure command of the map. ```vim nmap <C-s> :w<CR> nmap <C-p> "+p vmap <C-y> "+y ``` The above commands are pasted in the vimrc file. These keymaps once sourced will stay in permanently unless you remove them from the file. ## Keymap Modifications We can also add extra arguments to the keymaps such as noremap and remap and others. In remap, the keymap is recursive which means the key pair will get mapped to the keys mapped already somewhere. The keymap will redefine its map depending on the already existing maps. It is preferred to use noremap as it will not mix up the existing maps and have a fresh key binding. The mapping in Vim is quite versatile and there are quite a lot of modifications you could do to make your key binding feel and work according to you. So for further un depth understanding of modifications of keymap in Vim you can check out their documentation [here](https://vim.fandom.com/wiki/Mapping_keys_in_Vim_-_Tutorial_(Part_1)). ## Keymapping Usecases Now, it's entirely up to you to make keymaps according to your preference and choice. You can use your creativity and knowledge to make powerful and efficient keymaps. For some insights, I'd like to give in some cool and productive keymaps which I personally use very frequently in coding a particular programming language. ```vim nmap <C-s> :w<CR> nnoremap py :!python %<CR> " Python run macro nnoremap cpp :!c++ % -o %:r && ./%:r <CR> " C++ compile and run macro nnoremap sh :!chmod +x % && source %<CR> " Bash sourcing macro nnoremap c :!clang % -o %:r && ./%:r <CR> " C compile and run macro nnoremap jv :!javac % && java %:r <CR> " Java compile and run macro ``` The above mapping will run the file or compile and run the file you are currently editing in Vim. That is just an overview of how you can use key mappings in Vim for any programming language or tool. Vim allows you to modify and make the keys do the heavy lifting very easily and save a lot of time and frustration. This is how you implement DRY(don't repeat yourself) perfectly. I personally think this is just perfect to do certain things which are quite common and sometimes daunts you to do this again. So just have a look and play around with Vim keymaps, it is the power with which one can excel in saving time and energy. This is just flawless. OK, That was too much from my side. ## Keymapping and Macros Some Keymappings provided above are quite interesting for a person trying to understand macros and key binding. A macro is a bunch of commands packed in together to access it very efficiently, Not the exact definition, but still, it just means to simplify things and saves time. Keymapping is the same thing but to map with the keys. From some snippets and explanations of VIm keymaps, it must be easier now for a beginner to understand Keymappings in Vim and how to customize accordingly. I hope it helped, Thank you for listening to my understanding of Vim keymapping. Below are some of my keymaps currently in vim for windows. Some of them were already shown in the code snippets.  So, Keymaps are quite powerful and can be a bit difficult to set up in a single try. Keep experimenting with keymaps and make Vim the powerhouse of productivity and customization. Happy Viming :) https://www.meetgor.com/posts/vim-for-bash Vim and BASH? Bash Scripting is a powerful skill to have as a programmer because we find Linux almost everywhere and to get through it you must have a command o Thu, 10 Jun 2021 00:00:00 UTC ## Vim and BASH? Bash Scripting is a powerful skill to have as a programmer because we find Linux almost everywhere and to get through it you must have a command over its interface which is generally the BASH shell. Vim is a great option for doing this, or probably the best out there! Because Vim is pre-installed in almost every Linux distribution. This is not an in-depth setup for BASH on Vim, it is a simple editorial on starting up BASH scripting on the Vim editor. So without wasting time on "Vim features" let's dive in with the setup for BASH in Vim. ## Boilerplate macro Setting up a bash script doesn't require much code but still in some cases it can be a bit hassle and to avoid the repetitive task, one can easily set up a macro for the boilerplate BASH script. ```vim nnoremap bs i#!/bin/bash/<ESC>o ``` Ok that was pretty dumb but it can scale pretty quickly and it will be nice to tailor it as per needs, here's some snippet with function pre-loaded. ```vim nnoremap bs i#!/bin/bash/<ESC>o nnoremap bs i#!/bin/bash/<ESC>o<ESC>ofunction main(){<ESC>o<ESC>o}<ESC>ki<S-TAB> ```  When the key bs is typed in normal mode, you enter into insert mode as per the command macro, then we type in the required text and escape to move to the next line and continue the same stuff. This could be extended further like making some input or printing out some text and any other formatted text that you could think it as repetition. ## Sourcing Scripts So, after creating the file, sourcing the script, and running it can be a bit slow for some people, as you have to go to the terminal and toggle in the permission to run the script and then run, But pull on your seatbelts as this is VIM! You can die due to slowness! ```vim nnoremap sh :!chmod +x % && source % ```  When the sh keys are typed in the normal mode, the preceding command after ! (bang) will be executed in the terminal, the && keywords will execute the second command only when the first command is successfully executed. That just can so fast! Imagine doing this for long scripts and especially for debugging, it will waste 2 minutes every time you leave the editor and for 10 times you do the debugging, you will carelessly was roughly 20 minutes! Improve your debugging skills surely :) ## Plugins There are very few plugins out there for BASH as for VIM, but it's quite to write scripts even without any plugins. One of the most supported and popular plugins for BASH in Vim is [Bash-Support-Vim](https://www.vim.org/scripts/script.php?script_id=365) for auto-completion and [Shell-Check](https://www.shellcheck.net) for finding/correcting any bugs or error in the script . The mentioned plugin is quite awesome and it can greatly improve the speed of scripting for BASH, some commands such as shortcuts for writing if-else, while, for loops, commenting and other aspects in the scripting. The thorough documentation for such commands is also provided by the [plugin website](https://wolfgangmehner.github.io/vim-plugins/bashsupport.html). This can be used for autocompleting keywords and writing nested if-else and other logical operators in BASH scripting. Again, you can do absolutely fine without plugins in Vim as it is heavily customizable to the user's need and can be very rewarding to set up your own configuration for BASH. You can use standard Vim(barebones) for auto-completion as well with the command CTRL+N and CTRL-P to move down and up respectively. ## Some More Tricks BASH in Vim can be quite versatile to use as it provides some custom addons to make the script more functional and easier to understand. Some tricks such as using autocompletion can be quite inconvenient to use at once but it can get really smooth after some runs at writing the scripts. - In BASH Scripts there are quite a lot of brackets to play with that's why to jump around swiftly around such parentheses or brackets you can use **% to move from the opened to closed brackets or vice versa**. - You can execute any terminal command from Vim, be sure to be in command mode and press ! after the command, you would like to execute. This will run the command from the terminal and you don't have to leave the editor, it saves a ton of time and it's blazingly fast. - With the above trick, you kind of have a superpower within Vim to make, build, source, run the files or scripts within Vim, that is not repetition but it can run bash within bash. Ok! that's was pretty fast. Don't die of quickness now! Writing BASH scripts in Vim can be also boosted by using some built-in commands such as adding comments for multiple lines at once and some unexplored stuff which can be learned in the way to understanding the flow of Vim and BASH together. Happy Coding and Viming :) https://www.meetgor.com/posts/vim-for-python Introduction Vim is quite a powerful text editor which can add performance to the already fast typed language Python. Vim can be highly customizable and efficie Sun, 06 Jun 2021 00:00:00 UTC ## Introduction Vim is quite a powerful text editor which can add performance to the already fast typed language Python. Vim can be highly customizable and efficient to use as it has the power of **adding custom plugins and plugins managers, key mappings**, and the most critical weapon of vim - Access to the terminal straight away. This is not a full-featured guide of using vim for python, it's just a quick setup for using python on vim blazingly fast!! ## Plugin Managers So let us start making Vim, the text editor suitable for a python programmer. Firstly we'll need the vim plugin manager. There are different plugin managers out there, each of them has the same purpose to install, upgrade and manage the plugins for vim. You can install any one of them and get up and running. - [Vim Plug](https://www.vim.org/scripts/script.php?script_id=4828) - [Vundle](https://github.com/VundleVim/Vundle.vim) - [Pathogen](https://github.com/tpope/vim-pathogen) These are some of the finest and well-supported plugin managers in vim. You can use any of these plugin managers, and get started by installing some plugins. ## JEDI-VIM- Auto completion Firstly I will like to install Jedi for code completion in Python. The plugin can be simple and straightforward to install using any of the above plugin managers. Jedi-Vim provides some neat and clean** syntax analytics and autocompletion for Python in Vim**. You'll find the docs and installation process here [JEDI-VIM ](https://github.com/davidhalter/jedi-vim) ## NERDTree-File manager Next, It would be great if we install a file manager for managing the files and folders in the code directories. We can simply use the Nerdtree plugin for this. NerdTree is quite a **fantastic plugin for file management in Vim**. It simply makes Vim feel like VS Code. The installation and docs can be found here [NERDTree](https://github.com/preservim/nerdtree). Nerdtree commands can be longer to write, for that let's start mapping and for that, we can start editing our Vimrc. ```vim set number syntax enable filetype indent on set tabstop=4 set softtabstop=4 set autoindent set encoding=utf-8 ``` This can be some addition to your existing vimrc as you might have a setup for plugin managers. You can choose the color scheme of your choice, don't waste time selecting the color scheme. Feel free and modify the vimrc according to your knowledge and choice. ## Keymappings We move on to the Key mappings for NERDTree and other features. You can make mappings generally in the normal mode but there might be cases where you need to use maps for visual mode or insert mode as well, that entirely depends on the user :) To map in normal mode, we'll its command to be specific: ```vim nnoremap <C-n> :NERDTree<CR> ``` This will map CTRL+n to open the NERDTree file manager to the left, saving a bit of time and avoiding frustration. Feel free to add any keymap of your choice, this is just for demonstration. You can further automate NERDTree for switching between tabs because it makes you type CTRL+w twice, you can reduce that to just typing w. ```vim nnoremap w:<C-w><C-w> ``` ## Integrated Terminal Macros We can open a terminal window like a split between the editor. We can simply use the command :terminal to split the window horizontally, where the upper split will be terminal and the down window will have the editor. This is quite a neat feature of Vim in that it blends with the terminal so well so that we can switch between the terminal and the editor very quickly. For that, you can create a macro if you need to fire up a terminal again and again. ```vim nnoremap <C-t> :terminal<CR> ``` If you place the above macro in your vimrc and then type Ctrl+t, the exact thing will happen to fire up a terminal split but with fewer keystrokes and without leaving the normal mode. Also, the NERDTree macro can be also fruitful with this as it will make a full-blown IDE-like feeling inside of Vim.  ## Running the Code with a snap We can automate the process of running python scripts inside of vim. Instead of typing out the entire command for executing python script from vim. We can use keymaps for it as they can significantly boost the time required to run and debug the code. ```vim nnoremap py :!python % ``` This is a small map but can save a lot of time and give some motivation to use vim as you run the code blazingly faster than other editors. I have used py, but it can cause some problems as p is already mapped for pasting. So it's better to use other key combinations such as ty, yh, or any other key combination of your choice. Try it out and add your own flavor that's how we all learn. So, that's the basic set-up for python on vim, you can make more custom mappings, find more plugins and test out which work out the best for your workflow. Happy Coding and Viming ;) https://www.meetgor.com/posts/vim-text-editor-ide We all are asked to use visual studio code and other rich looking editors as a beginners text editor or an IDE but that makes a habit of a rich and pleasing int Sat, 20 Feb 2021 00:00:00 UTC We all are asked to use visual studio code and other rich looking editors as a beginners text editor or an IDE but that makes a habit of a rich and pleasing interface of running and debugging programs, and that is the reason we don't feel good to use command line or the terminal. Terminal or Command line are quite important to learn and are powerful as well. Vim is a text editor, not any text editor but a special and one of the fastest out there if you use it wisely. Vim can be used in the terminal which means it can run terminal commands inside the editor interface. That makes it quite a great option to improve your terminal command skills and also for enhancing your Linux knowledge. You can run program files from vim itself just with a few keystrokes. This text editor is preinstalled in most Linux distributions and hence you could quickly edit some files without going anywhere. So that being said, vim is not easy to learn, you have to put some effort to reap the fruits of saving time in the longer run. It takes time to get used to the commands and key combination of Vim. ## Download VIM But, let us get our feet wet in learning a few commands in Vim. To download vim you can go to the official [Vim website](https://www.vim.org/download.php) . Firstly let us understand how to open vim from the command line or the terminal If you are on windows you have to add the vim.exe file's path to the environment variable PATH. After that open CMD or terminal for windows 10 and type vim and enter. You are in Vim. If you are in Linux or macOS, then open the terminal and type vim and enter. After you are in Vim, you will surely be trying to type something right? else you are not a geek ; ) ## VIM Basics You won't see anything happening even if you are typing. This is because are 4 basic modes in Vim, understand it like a room. The modes are 1. **Normal Mode** 2. **Insert Mode** 3. **Command Mode** 4. **Visual Mode** Wait for a second, you should learn how to move in vim first but you cannot learn everything at once, So take it bit by bit. To navigate around vim like blazing fast you can use h,j,k, and l. I know it's wired but you will get used to it. **h** -> move left. ( hop left) **l** -> move right. (opposite of left) **k** -> move up. ( keep it up) **j** -> move down. ( jump down!!) If you want to apply certain movements several times you can use numbers before hjkl to move the number of times in that direction. Let's say you type 23j , here you will go down 23 lines. You can also use : and number to jump to a particular line of that number, :12 moves you to the 12th line. The default mode is the **Normal mode** where you can navigate in your file, like a passageway between different rooms. Each other mode is accessible via Normal mode. You have to enter normal mode before switching to the other three modes. By default, after opening vim you are in Normal mode. You start inserting text in the file while being in Normal mode but you can replace text and edit the existing text in the file. **Press ESC to enter Normal mode**. The main mode in VIM is **Insert mode**, where the actual typing and text editing takes place, **Enter i to enter insert mode**. After entering i you can type anything you want. After you feel satisfied with writing in VIM, you can escape out of the insert mode (room) and enter the normal mode (passageway). Now the biggest question VIM beginners ask **"HOW ON EARTH SHOULD I EXIT OUT OF VIM ??"** . Take a breath dear, you need to save your file first. To do that let's explore a different mode. The next crucial mode is **Command mode**, where you have to remember some commands to do something. Let's continue from where we left. To save our file, **Press : and Voila!! you are in Command mode**. After it depends you want to save the file, quit vim, save the file and quit vim or quit vim without saving the file. OK, that is too much to take in one go. Be sure to be in Command mode to press the following keys (i.e. press colon before any keys if you were in normal mode). **w** -> save file. **q** -> quit vim. **wq** -> save file and quit vim. **q!** -> quit vim without saving the file. These are a few of the commands available in VIM :) So now let's talk about **Visual mode**, the little helper mode (room) to do cut/ copy in VIM. There are many ways to edit or add text in Visual mode. You have options to choose from character by character, line by line, or block by block. **To enter Visual mode press v**. This gets you into character by character highlighting whether you want to cut the text or copy (yank) the text. After selecting the portion you can either cut that portion or copy it in vim register. **y** -> Copy the selected portion. **d** -> Cut the selected portion. You can use **V (Shift+V) to enter Line Visual mode**, this selects or highlights the portion line by line and not single letter by letter. You can also use **Ctrl+V to enter Block Visual mode**, this selects a block of code. Here is a good way to remember switching between basic modes in VIM ->  Switching between basic modes in VIM It seems to be quite slower than VS Code, doesn't it? Ya, I know it's kinda slow but with time it picks up pace, just like a train. Spend time in VIM and it will be worth it. No Text editor is so powerful as VIM, it is so for a reason. If you are curious to dive deeper just dive in fully or stay in VS Code and your fancy. There is a great community around VIM and you'll be happy surely to be a part of it. Oh! The quick fact even VS Code has an extension for VIM ;) ## Some more Basics You can do most of the basic programming using this but to be a bit faster you can use some tweaked commands to do stuff like, **I** -> Enter insert mode to the character at the beginning. ( i puts you in insert mode to the cursor) **a** -> Enter insert mode adjacent right to the cursor. **A** -> Enter insert mode at the end of the line. **o** -> Enter insert mode a line below cursor. O -> Enter insert mode a line above cursor. These commands are an optimized way to enter insert mode in a suitable way. These commands can be used as per needs and quite significantly improve editing speed. You can also shift from one mode to Normal mode by pressing Ctrl+C or Ctrl+{. Whatever you feel good with. Pasting is a programmer's everyday task. No, it's good as a beginner and only if you are not running as it is. In VIM you can paste from your previously copied text using Visual mode or deleted chunk just using p to paste. Remember if you are on a character it will paste on starting from where your cursor is, which means your character on the current cursor will be overwritten. You can also use "0p to paste from the copied buffer. If you are pasting from an external source i.e. out of VIM, you have to use "+p . It pastes from the next character from the cursor. If you want to search a word in a file, you can use navigation commands like hjkl but there's a faster way around, If you press / and type the word you are searching for, you will get the first instance of that word. If there are multiple instances of the same word, you can press n to got the next instance. You can also press Shift+N to move to the previous instance of the word. Some more word searching ahead. Say if you are already on the word which you want to find its instance, then you could press * to move to its next instance and # will do the same backward. Oh ! that was smooth. You can see that we are just using our keyboards while using vim. This is a great feature of old but gold text editors like vim and others that make you faster on the keyboard and avoid mouse traps. If you want to jump to the end of the block or a paragraph you can use { and } to go to the beginning and the end of it respectively. **G** -> to the end of the file. **gg** -> to the beginning of the file. **L** -> move the cursor to the end of the screen. ``` :%s/old/new/g ``` in Command mode -> replace the old word with a new word throughout the file(g) Ctrl p in Insert mode -> auto-complete reference from current file. So these are some quick and pretty good commands to get started with and feel VIM. But this is just VIM used 25% efficiency, you would have to go to the command line or terminal to run and compile the program you have written. It could waste a lot of time. But as a beginner, this is pretty decent and helps to learn the core programming, behind the scenes of compiling and learning new stuff and commands. It gets pretty exciting if you have got the basics cleared. But be sure to make it through. It's time to use VIM to its actual powers. VIM + terminal. It can get quite fast. So beware!!! # Running Programs from VIM It turns out that you could go to the command mode and run terminal commands from there just use! before the command first. Linux/Mac Users: use terminal from vim ``` :!ls ``` //-> ls command from vim Windows: use CMD from vim ``` :!dir ``` -> dir command from vim You could also use other commands which are appropriate to be executed from vim. Here is where the trick comes in, you don't have to quit vim now to compile or run the program. Let's say you are writing a C++/C code and you have a clang compiler. Then you could use c++ and c respectively command with appropriate file handles to compile from vim. Here's the command, ``` :!c++ % -o %:r ``` Here, % stands for the current file and :r removes the extension of the file. So we are creating an executable file without the extension and hence :r. And to execute the file, simply type, ``` :!%:r ``` this will execute the program. If Windows users face an issue like a file is not identified as a batch file or executable file, you should add .exe after :r to make it an exe file. For Python or other interpreted language, it is quite simple, just the interpreters name with the file name as it is ``` :!python filename.py ``` ``` :!python % ``` You could find your preferred language's CLI tool and set it up for your environment and get it plugged with VIM. **Mapping in VIM** But this is not it! There is a long way to master VIM. It's not possible to know everything about anything. So for getting faster I'll introduce VIM's fastest tool and the part where it really shines. Mapping!! You could map (assign) some keys to do a particular task in different modes. Just imagine typing cpp and running the program or py and running the program,no need to go to the mouse to click run button and close the window. It gets really blazing fast. Only your creativity is the limit. Mapping commands are also called macros. ``` nmap cpp :!c++ % -o %:r && %:r ``` ``` nmap py :!python % ``` In the above commands, nmap means in Normal mode MAP cpp and then followed by the command which for this case it is executed in the command mode. There can be various permutations you could make like imap or vmap in Insertion mode and Visual mode and so on. From mapping,its basically left-hand side mapped to right-hand side. The key combination to the left and command to be executed to the right. You will surely say, "Do I have to do it every time I run the program or every time I use VIM?" Well, You can :) but you should not. Because there's a way around it and this is the fuel of VIM or the soul for its Speed. Yes, I am talking about VIM CONFIGURATION FILE or vimrc # VIM Run Commands File(vimrc) This is really a nasty feature of vim. You could tailor your vim editor as per your needs. You have to write the command once and it will run from the vimrc file. Making a vimrc file is a bit overwhelming in the beginning so, you should have a concrete knowledge of VIM commands and you should remember commands as well. - Make a file called ~/.vimrc - You should be using VIM to edit it or write it of course! Type vim ~/.vimrc in terminal or cmd - Enter some commands such as set file type indent on , set number , you can find a [sample vimrc file ](https://vim.fandom.com/wiki/Example_vimrc) on google and understand its commands and uses. Paint your plain canvas vimrc with your own creativity. From here you can add the mappings and other key combinations that you might have been using while learning VIM to your vimrc. VIM also has plugins and more customizable tweaks that can enhance your productivity. You could always get more help in VIM by using :help commandname. Also you can learn from vimtutor from terminal or CMD. If you want to dive in more deeper , i link some few resources and video links to get your vim skills to new level. [VIM Official Documentation. ](https://www.vim.org/docs.php) [VIM fandom Wiki](https://vim.fandom.com/wiki/Vim_Tips_Wiki) [130+ VIM commands. ](https://catswhocode.com/vim-commands/) [Vim Interactive Guide.](https://scotch.io/tutorials/getting-started-with-vim-an-interactive-guide) [Vim as your editor.](https://www.youtube.com/watch?v=H3o4l4GVLW0&list=PLm323Lc7iSW_wuxqmKx_xxNtJC_hJbQ7R) Vim is a tool that can get quite handy for programmers and it stands out from rest of the text editors. By using Vim you will be in very rare people who can make something from almost nothing. Because in VIM you make everything from commands to mapping from customization to functionality. This cannot be done overnight for sure and you would say you are wasting time in the initial stage but no, you are actually building a foundation for more efficiency and speed. If still you want to use VS Code ,you can use it freely . Because it won't matter much , at the end of the day its programming skills that win the race. **This is quite a huge topic and everything could not be explained in a article, it should be self-explored and learnt from various sources to get its best functioning as per needs. This was a quick guide how and why you should be using VIM as a text editor or as a customizable IDE. As you might have seen its functionality is quite customizable and it is quite powerful tool for a programmer. Happy Coding and VIMing ;)** https://www.meetgor.com/posts/django-database-conf Introduction In this part of the series, we will create an external database and configure the settings for that database. We also look into the process of migr Mon, 01 Jan 0001 00:00:00 UTC ## Introduction In this part of the series, we will create an external database and configure the settings for that database. We also look into the process of migrations which is to convert a model which we created in the previous part into the actual structure/schema for the database. I'll be using `PostgreSQL` for most of the demonstrations but it should be similar for other database management tools. This part will make your Django project link to a local database. ## Selecting a Database If you have read my previous article about models, you would have got a glimpse of the tools to manage a database, but not quite to choose a database for your project. This section covers that specifically. We will see what options you have while selecting a database. - SQL databases - sqlite - PostgreSQL - MySQL - MariaDB - Oracle Selecting a database hugely depends on the type of application you are going to make, but most of the time it's SQL as a choice just because it has been dominated for a variety of application types over 4 decades. Still, NoSQL is growing in popularity and has some advantages over SQL in many modern applications. You need to analyze your project a bit deeper and understand the flow of data in a better way to make a decision about SQL and No-SQL, but most of the time it's gonna be SQL. **Also Django doesn't officially support NoSQL, so you'll have to turn up some third-party libraries to integrate and manage a database.** After deciding the type of database, you have one more decision to make here. It's picking up a DBMS tool. There are a lot of databases like PostgreSQL, MySQL, MariaDB, Oracle, etc. you need to pick whichever you feel comfortable and the one which suits your project architecture and requirements more closely. Though there might be very few differences in all the SQL Database tools there are a few things that distinguishes one from the other. ## Creating a Database To create a database, you need to go to the Admin app of the DBMS tool you are using, for Postgres it's pgAdmin, for MySQL it's MySQL Administrator or PHPMyAdmin. You need to do the research for setting up a database locally for your project. But Django already is paired with `SQLite` which is a relational database but with a few quirks. It is really great to get started with a project without creating or managing an entire database system. The SQLite database is all contained in a file called `db.sqlite3`. If you want to work on a particular database like PostgreSQL, MySQL, etc. you need to create the database using the management tool and keep the configuration data like `name`, `host`, `password`, etc. after creating the database. I'll give a demo of creating a simple database in PostgreSQL but mostly it is a bit different in other DBMS tools as each of them have their own GUI applications. Install [Postgres](https://www.postgresql.org/download/) and [pgAdmin](https://www.pgadmin.org/download/). This is a demonstration of creating a database in pgAdmin -3  This is how you create a database in pgAdmin-3, it should be quite straightforward and simple to follow in other DBMS tools as well. You will also require a password to open the Admin interface for these tools, so keep that password handy we'll require that later. The following is the process to create a database/schema in MySQL  This will create an empty database in your local machine. We'll want our Django project to link that particular database and use it inside its applications. In the next part, we'll configure the `settings.py` file to access the database from the local machine. ## Configurations for Database We need to configure the `settings.py` file for integrating the database in our project. Django has a dedicated section for the database in that file. By default, the database configuration is created for the SQLite database which as I said earlier it is the default database that Django uses if not specified and configured. So, when you first see the configuration for the database in the `settings.py` file, you will see configuration like the following: ```python DATABASES = { 'default': { 'ENGINE': 'django.db.backends.sqlite3', 'NAME': BASE_DIR / 'db.sqlite3', } } ``` For PostgreSQL or any other database, we will require more things than the `sqlite` configuration. Namely, we will need the following: - `name` of the database - `user` of the database - `password` for that user - `host` and `port` for the database. The port is optional as most of the database tools pick up the default port for their purpose. For PostgreSQL, the default port is `5432` and for MySQL is `3306`. Leave it blank like `'PORT': '',` if you are not sure what is the port for that database. We also need to specify the `ENGINE` as it is the database backend to be used for the project. There are options for specific databases as mentioned in the [documentation](https://docs.djangoproject.com/en/4.0/ref/settings/#engine). **PostgreSQL**: In PostgreSQL, the default user is `postgres`, it might depend on your configuration and setup though. The pgAdmin will prompt you for a password when first installing the Postgres on your machine. The password to be used is for the default user of the database. The `name` is the name that you gave while creating the database in the Postgres admin section. Finally, the host is generally `localhost` as we are using the local database which is our system, and `port` as said earlier is `5432` by default for PostgreSQL. ```python DATABASES = { 'default': { 'ENGINE': 'django.db.backends.postgresql', 'NAME': 'blogapp', 'USER': 'postgres', 'PASSWORD': '12345678', 'HOST': 'localhost', 'PORT': '5432', } } ``` **MySQL**: For MySQL, the default `user` is `root` and the `port` is `3306`. The password is the default password you use to access the MySQL Administrator application. ```python DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'blogapp', 'USER': 'root', 'PASSWORD': '', 'HOST': 'localhost', 'PORT': '3306', } } ``` For further details on how to configure a Database in a Django project you can check out these articles: - [PostgreSQL setup in Django](https://dev.to/mungaigikure/how-to-set-up-postgres-in-your-django-project-575i) - [MySQL setup in Django](https://medium.com/@omaraamir19966/connect-django-with-mysql-database-f946d0f6f9e3) - [Maria DB setup in Django](https://medium.com/code-zen/django-mariadb-85cc9daeeef8) - [Mongo DB setup in Django](https://www.mongodb.com/compatibility/mongodb-and-django) MongoDB is a NoSQL database so, it will be quite different to set up and configure the database at least in the longer run. I've no experience with NoSQL so please forgive me in this part. There are very rare instances you will need a NoSQL database with Django. There are other configurations as well which might be specific for the purpose and can be explored in the [django documentation](https://docs.djangoproject.com/en/4.0/ref/databases/). ### Verify the database connection To check if the database was actually linked in the Django project, I'll introduce you to a great tool in Django: `python manage.py shell` (make sure to be in the virtual environment). This command will open a python interpreter in the shell. It is an interactive console so that we can test some aspects in our project. For instance to check if the database is connected or not: Execute the code after running the command `python manage.py shell` from a virtual environment. ```python import django print(django.db.connection.ensure_connection()) ``` If this returns `None` you are good to go. And if the result is tons of error messages, you have something wrong in the configuration or the database itself.  ### Installing database adapter Before we can do anything with the database, we need one thing which is a **database adapter**. Now, this is dependent on the database you are using. The list is as follows: - [psycopg2](https://pypi.org/project/psycopg2/) for PostgreSQL - [mysqlclient](https://pypi.org/project/mysqlclient/) for MySQL and MariaDB - [cx-Oracle](https://pypi.org/project/cx-Oracle/) for Oracle SQLite does not require an adapter as the database is a file stored in the base directory as `db.sqlite3` The above list is nothing but Python packages that allow the Django ORM (which is under the hood python) to operate the database. To install them you can simply install with `pip` as `pip install psycopg2` , `pip install mysqlclient`, and so on. Make sure you are in a python virtual environment. After installing the particular package, we can now move on to the migration process. ## Migrating the models Now, we have a clean and fresh instance of a database created. What next? We'll now use the logic in the [previous part](https://mr-destructive.github.io/techstructive-blog/django/python/web-development/2022/01/04/Django-Basics-P7.html), where we created and designed the logic for the database i.e. `Models`. Now, we can combine our physical database i.e. the database we just created on a DBMS tool locally, and the logical model to populate and give it a structure. We'll perform **migration** on our database. This is where the magic happens and probably the step which should be carefully be executed if working with a real-time (production-level) database. Because the commands you'll run are gonna directly affect the database. Making a migration is a two-step process. But what is migration? The Django documentation states it as the version control for the database schema and takes their word for that. It is basically a folder(hidden) that stores the state of your database structure just like commits in git. Let's see it practically ### Makemigrations For actually creating tables, relations attributes in a database from a model. We use the command `makemigrations`. This command creates a file inside the `migrations` folder inside the application folder whose model has been recently created or updated. It doesn't affect the database but it creates a file which in turn after a `migrate` command will be parsed to the actual database using the ORM. So, for any attribute or logical change inside the model we use the `makemigrations` command as below: ``` python manage.py makemigrations ``` We do not use the above command if any functional change i.e. operations involving querying the database and other operations that don't affect how the database is structured or stored. Though we have to use the `makemigrations` command when the fields in the model are changed even slightly. ### Migrate To see the result or create the actual tables, attributes, and relations among the tables in the database, we need to run the command `migrate` which will see the latest file in the migration folder and execute the queries to change the schema of the database. So, this is a very powerful command that can perform SQL queries under the hood with python. The below demonstration shows the output of these two commands in a Postgres database. ``` python manage.py migrate ``` ### Demonstration  Let's follow the GIF/video in sequence: 1. Create the database in the DBMS tool 2. Configure `settings.py` for the database connection 3. Install `psycopg2` or DB-specific adapter. 4. Create the model. 5. Run `python manage.py makemigrations` 6. Run `python manage.py migrate` (if the above command is a success) Here's the model from the demo: ```python class Article(models.Model): title = models.CharField(max_length=255) post = models.TextField() created = models.DateTimeField(auto_now_add=True) updated = models.DateTimeField(auto_now=True) ``` If we have multiple apps and we only want Django to migrate a model from a specific app, we can simply add the name of the app to the command. By default, Django will migrate all the models from the `INSTALLED_APPS` list so we need to specify which app to migrate explicitly. ``` python manage.py makemigrations app_name ``` This also applies to the `migrate` command. To understand the migration process more deeply, let us see another demonstration of changing the model and then applying migrations. We'll keep an eye on the `migrations` folder inside of the app in the project.  We can see that initially when we applied the migrations in the previous demo, there was a single migration file called `0001_initial.py` but after we altered/changed the title's `max_length` from 255 to 127 and executed the `makemigrations` command another file called `0002_alter_article_title.py` was created that only contained the changed field. This is very similar to git diff in which we are about to commit to the actual database. So, that was all about migrations. There is definitely a lot to be covered about migrations and databases, but I'll leave you here for now. There are a lot of things to understand and learn before we can talk more about the databases. We now have a base for other things to get our heads around. We now have a database linked to our project, still, we don't know how to use fetch or query them. That is a topic for another part. We didn't use SQLite database which is by default provided by Django because in production it can just listen to one request/query at a time. That is not ideal for most web applications. Still, there are use cases of this database as it is suitable for embedded database systems for storage applications in desktop/android applications. ## Conclusion So, from this part of the series, we were able to configure a database more specifically an external database (not the default SQLite DB) in a Django project. The concept of migrations was understood and demonstrated with live examples. Also, the process of creating and designing models was used from the previous part to create the structure in an actual database. In the next part, we shall dive into the Admin Section that Django provides to view and tinker with the local database. If you have any queries(not database queries) then please let me know, thank you for reading, and until then Happy Coding :)