Meet Gor - Tag: python

Flight Observatory Mumbai Airspace Case Study

Thu, 02 Apr 2026 00:00:00 UTC

## Introduction I do write blogs, newsletters, but this is something wild I had written in a while. I was always curious and interested in knowing about the flights flowing over Mumbai, I do always look up and see the Airline when it appears above my head in front or on the side. But I never actually read or discovered anything true or factual about it. As a software developer, that is very unsettling, I have some ideas and observations but never tested them. I can excuse myself that I am busy and that is a toy little project to get off my hands. But now, we have LLMs and no excuses. I don't need a second invitation to explore my curiosity, off I went. And came back with around a week or 2 of digging in data, downloading the ADS-B stream of airspace and crunching the numbers for the BOM (VABB) Mumbai's Chatrapati Shivaji International Airport (CSMIA) and its vicinity. The numbers I must tell you were staggering and I couldn't stop starring at them, I felt relieved and felt my childhood was worth while. I am writing this blog post to explain to myself, how I got here. It was a chaos in start, total misery of data. But after a few attempts of successfully accumulating the data only for BOM airport's vicinity of 300 kilometers, I was able to finally make sense of the pile of data (since at one point it looked like a goble of mess). This will be a two part blog. This one being non-technical (that is no grilling from peer developers) which will explain the why and what of the case study of the Mumbai Airport. The next part (out in a couple of days from now) will be a treat for developers (or curse words for myself, ok it won't be that bad), which will explain the how. This part is about my intuitions and the memories I had since growing up near the vicinity of Mumbai Airport. I explain what dataset I used (it was not a simple available dataset in the public, mind you that), how I structured it, what questions I asked, how I formed the queries (not in terms of SQL but human logical terms), and how I reason through the results. And trust me everything made sense. So, if you have been in Mumbai, and wondered how many planes flew above your head in a day, you are in for a treat! Buckle up! Because Mumbai Airport never stands still. Check out the full case study here -> [Flight Observatory - Mumbai Airspace Case Study](https://dev.meetgor.com/flight-observatory/case-study/mumbai-airport/) ## Flashback of Mumbai from 2010-2018 I have spent my life in Mumbai, the two decades that I have lived here, I often feel nostalgic lookin back at the memories. The one that stands out even today is the plane spotting. Since childhood, I have seen, heard, observed planes flying above me, beyond the buildings, buzzing and slashing the concrete jungles and flat blue slums. I distinctly remember the days from my school, the 7th A classroom, located on the top of the school, the highest room on the school. I remember the days I shared sighting planes through the periods with my friends. It was a nostalgic experience to see - the regular "Air India" on time, - "Lufthansa" with its guzzling sound, - occasional "UPS" with the jumbo cargo, - and the millions of "Jet Air" and "Spicejet" among them. There was a different joy of just seeing and observing them, admiring the beauty of Boeing 787, 777 and 747s, Airbus A330 and occasional very rare Airbus A380 the king. Those are fresh in my mind. I was lucky to be honest to have an airplane enthusiastic friends with me. One of them was notoriously a die-hard fan of British Airways though. The 11:45 British Airways was a treat to him, with me cursing words at British people, hoping the plane to crash (just teasing him of course). We almost had a anticipation on the next plane to come and over the year we had a schedule that which one will come next depending on the period at the day. We had formulated that the BOM Airport's frequency was 2:30 minutes (that's 24 flights in an hour, 576 in a day). That is a lot of flights landing over our head, I used to brag that BOM was (is?) the single runway busiest airport. And take a strange pride in defaming British Airways. Those were the times :) And now, I don't have that luxury to sit with that friend and gaze at the sky (if I will, I might get existential crisis I think). I wanted to get those moments back, I don't have the luxury to sit with that friend though, but I realize I have another friend a tool of sorts. LLMs. I realize I am a programmer now, whatever I want to do, I can. Its just a wish away. ## The Idea And I began making this project. To observe the sky. Not by my eyes. By tracking, analyzing and crunching data from APIs. Nothing comes for free. Constraints start developing, but there is no engineer who has no constraints. For me I didn't want to setup a cloud infrastructure(not that I can't I just don't have the budget for it) and do bunch of credit card swipes, I didn't have the luxury for that, and for buying heavy 200$ subscription of those AI-tools either. So what? > Engineers aren't made from freedom. They're shaped by constraints. I explored every API out there to see what I can use the best. And lo and behold I found some. Not everything was great, there were samples, just a day for a month, limited data, but I kept digging and digging. I found what I needed, data points for Mumbai (or any airport for that matter) across a decade. I thought I might get full-day, full-year data, a decade worth of data. But what I got was a decade worth of data, for all months, but only for the first day of the month. Sigh! But I kept moving. I don't care how little the data is. It is real. It is 12 days of the year for 8 years. That is not a lot, but enough to find patterns. And I can surely say, Mumbai airport hasn't changed drastically over the decade. I thought that was not enough! I wanted more, I wanted live data. Can I get live? I though I could, I did but sort of. Its not the most accurate representation of flights. But a good guess, a practical one at least. So, I built [Flight Observatory](https://dev.meetgor.com/flight-observatory). A live and historical data analysis tool or a webapp of sorts for viewing flights captured by the ADS-Bs around the world. I kept storing the live data and gobbled them into a historical section. From the other dataset and some live observations(which turned out to be a disappointment) I also made a case study for Mumbai Airport. This will be not quite nerdy and technical, even if you are not a technical person, strap on. Its going to be a fun read. The next part will be a treat for the technical people reading this, so put on your glazing glasses and bickering comments for that post later. ## What I found (TLDR) Mumbai Airspace is very consistent. It is busy in a very Mumbai way: a lot of movement, but with a clear pattern. The traffic pattern is strongest in the early morning and evening. The daily activity spreads across the early morning, the workday, and the evening. In the airport-proximate analysis: - The strongest hourly movement shows up around 07:00 IST - Other heavy hours around 06:00, 11:00, 17:00, 20:00, 21:00, and 22:00. On the sampled days, the data produced: - `21,923` airport-proximate events - Split into: - `12,723` landings - `9,200` takeoffs - With a median landing-to-next-takeoff gap of `1.5` minutes. That last number matters. It is the airport's strength and the health checkup number. It says the airport is turning movements over very quickly, which matches the reputation Mumbai has for being tightly packed. The airspace itself is occupied for a while: - Median landing run: `34.5 minutes` - Median takeoff run: `25.1 minutes` So landings stay visible longer than takeoffs. That makes sense operationally, arrivals spend more time in descent and approach, while departures clear out faster once they lift off. The same aircraft also comes back around fairly quickly: - median same-aircraft ground time: `96.5 minutes` That means the aircraft are not sitting idle for long. The fleet is being reused in a steady loop, which fits a high-frequency airport like Mumbai. And the shape of the traffic is not random: - Saturday is the busiest weekday in the sample - the landing and takeoff corridors split cleanly across the airport area - the dominant heading clusters show the same directional axis being used again and again Across the full analysis set, I ended up with about - ~17 million observations (that looks like a gigantic lot, but it isn't) - ~151,000 inferred flight runs - Spread across ~4,900 aircraft hexes - ~4,800 registrations - ~140 aircraft types. That feels right for Mumbai. It is a city that never really stops moving (it actually sleeps, but internationally awake), and the airspace behaves the same way. The airport-proximate movement counts are also not tiny, it is structured, dense, and repetitive. The airport works like a tight system, not a loose one. ## Notes and Clarifications The data itself was not clean in the neat, airline-report sense. It is raw observation data: points every few seconds, over and over again, with altitude, speed, heading, position, and callsign changing as aircraft move across the sky every 5 seconds or 30 seconds(depending on the year). I had to group those observations into inferred flight runs before it became useful. Once I did that, the picture got much clearer. Most of the useful traffic is post-covid traffic too. The analysis is heavily weighted toward 2022 to 2025. > The ADS-B for the BOM (VABB) actually started the tracking of airports around late 2017, early 2018, and that too was per minute. So it was very sparse and not necessarily capturing the full spectrum. > This data is from 2018 to 2026, not the full day of each month. It is sampled to be the first full day of each month (i.e. 1st April 2024, 1st of May 2024, and so on). Also the data points from 2018 to mid-2020 is snapshot of 60 seconds and moving from late 2020 through 2026 (till march) is snapshot of 5 seconds. > The data is only for ~78 days(not continuous, its the 1st day of month). 2022-2025 full years (12 days), there are only 3 days in 2018, since ADS-B was active after mid-2018, and the dip in 2020-21 due to COVID, so little or no traffic observations. ``` ---------------------------- | Year | Number of days | ---------------------------| | 2018 | 3 | | 2019 | 11 | | 2020 | 5 | | 2021 | 9 | | 2022 | 12 | | 2023 | 12 | | 2024 | 12 | | 2025 | 12 | | 2026 | 3 | ---------------------------- ``` > The numbers are real, but they are still a proxy. They are not ATC logs. They are not airport ops records. They are ADS-B-derived movement patterns around Mumbai. That means the data is powerful for showing pattern, density, clustering, and dominance, but it should not be mistaken for official runway statistics. This data doesn't infer the activity the airport is doing exactly as in an official ATC sense, but it does tell you something real about how packed and continuous the traffic feels once you look at it through ADS-B data. > ADS-B short for Automatic Dependent Surveillance Broadcast that allows the aircraft to capture its own position and other details like (altitude, GPS based location, direction, speed, hex code i.e. unique aircraft code) and broadcasts (sends in the open as a radio signal), this gets captured by listening Air Traffic Controllers(ATC), Airports, satellites, networks, etc which then track and use it for internal purposes. That could be then used for tracking on the Airport, communnicating, averting accidents, etc. ## Initial Questions and Exploration I was looking for a historical dataset that could show me flight activity over Mumbai. I did not expect it to be perfect, but I did expect it to be workable. The first thing to understand is that this is not a clean flight log. It is a stream of observations. Each aircraft keeps broadcasting its position, altitude, speed, heading, and identity over time. That means the raw data is not `one row per flight`. It is more like `one row per movement in time for a given aircraft`. To make it useful, I had to turn those repeated observations into inferred flight runs. That is what makes the analysis readable. Instead of staring at thousands of raw position points, I can ask more human questions: - When is Mumbai busiest? - Which airlines show up the most? - Which aircraft dominate? - Where do planes start descending? - Where do they take off from? - How long is the gap between one landing and the next takeoff? That shift from raw observations to inferred runs is what made the rest of the project possible. I landed on the ADS-B Exchange historical data because it looked promising. At first glance, it seemed like I had found a long historical archive. But there was a catch I did not fully appreciate at the beginning: much of the data was sampled by month, and a lot of it was only the first day of each month. That was smaller than I hoped, but still useful enough to extract patterns from. That was the point where I stopped thinking about the data as a raw archive and started thinking about it as something I could actually analyze. ## Making "sense" of the data So, I had millions of observations. Because a single minute has 12 snapshots and if you do the math (not asking you to do it, but if you are curious), we end up with around 17,280 snapshot for a single day (12 snapshots per minute * 60 minutes * 24 hours). If you stack it up for around 12 months, that really grows big. So, what exactly were these snapshots? Could I directly use them in querying the database? No, not quite. It actually depends on what questions we might need answering. ### What is a Snapshot here A snapshot is basically a list of observations (or set of status) for the aircrafts currently in the airspace. The initial snapshot that I download from [ADS-B Exchange](https://samples.adsbexchange.com/index.html#readsb-hist) has a full world air-traffic and I never want that full data, I just want the tiny airspace around the Mumbai one. So we filter the latitude and longitude position around Mumbai. So, each snapshot finally becomes the aircrafts flying around the radius (30km to give a number) at that time. So we take those kind of snapshots at 5 second interval. So let's suppose there are 2 flights around Mumbai currently one landing and one taking off: These will be a snapshot. ```json [ { "icao24": "123456", "latitude": "19.123", "longitude": "73.123", "altitude": 500, "velocity": -100, "heading": 90, "timestamp": "2022-01-01T00:00:00Z" }, { "icao24": "8901", "latitude": "19.083", "longitude": "72.123", "altitude": 1000, "velocity": 300, "heading": 90, "timestamp": "2022-01-01T00:00:00Z" } ] ``` After 5 seconds you might see a change in the latitude and longitude (positions), the speed, and the altitude (also the timestamp). So we get another snapshot which shows the status of those aircraft currently in the radius of Mumbai, maybe in the next snapshot another aircraft is about to land, so it appears in the snapshot too. So on, you get the point right? If the aircraft stays within the radius of Mumbai for more than a minute (which everyone might, often more than a minute even 10-20 minutes?) then we might have 12 observations for that aircraft across those snapshots. We basically have a record or a progression of the aircraft over time. That is, in essence, a time series data. We are measuring a change of an entity over a period of time. So, this is what I get from the downloaded snapshots which are in millions. How do I make sense of them? Well, let's ask a question, then the scope can be reduced or we can move in specific direction. ## Questions on the data > What are the total number of flights over Mumbai in 2026 so far? Well, now you can see, we are talking about flights or distinct aircraft over a time. That is a distinct thing from the observations we have, we have to first lock in the time, but the deeper question is how do we group these observations to get flights? That is why the question need to change a little before it can be answered properly. I am asking how many distinct flight runs exist in the data, how long they last, when they start, when they end, and whether they are landing, taking off, or just passing through the Mumbai airspace. Once I do that, the raw snapshots stop being noise and start becoming something I can actually reason about. So, what constitutes a flight run? That is a question that we need to answer before proceeding. #### What is a Flight Run A flight run is my way of saying: this group of observations belongs to the same aircraft movement. It is not one row, and it is not one timestamp. It is the whole stretch of time where the same aircraft keeps showing up in the snapshots and its position, altitude, speed, and heading keep changing in a way that looks like one continuous movement. Now, how do we distinguish an aircraft? A unique aircraft means one distinct plane in the dataset, usually identified by its unique aircraft code, like hex or registration. This matters because the same aircraft can appear in many snapshots while it is in Mumbai's airspace. If I only counted rows, I would be counting the same aircraft again and again. But if I group those rows into a single run, then I can treat it more like one flight-shaped event. That gives me something much closer to what I actually want to study. ### Observations to Flight Runs So, I loaded the raw snapshots/observations into the dataset after compressing from json to csv to parquet files (dont worry about the details), into a duck database session. I had table schema like: - `observations` is just all the monthly Parquet files unioned together. - `observations_clean` cleans the fields and turns text like altitude, speed, lat/lon, etc. into usable numeric columns. - `observations_enriched` adds convenience fields like registration, aircraft type, airline_code, is_indian_registration, etc. From this we need to group into a flight run that could track each flight. So, I went and created a view that will do that: As we know we have `hex` in the data for each observations, that can be used to identify unique aircrafts in the airspace, however there is a catch. Each aircraft in ADS-B data is identified by a unique hex code, which represents the physical plane. But planes don't operate just one journey, they fly multiple routes throughout the day. This is where the callsign becomes important. A callsign represents the flight the aircraft is currently operating (for example, AI101 or 6E203). So, we need a combination of `hex` and `callsign` to identify a flight run. I sorted the observations by aircraft and time, and then compared each row with the one just before it. If the same aircraft disappeared for too long, I treated that as a break. If the flight code changed, I treated that as a new run too. That gave me a cleaner way to read the data. The same aircraft could still appear many times, but those repeated sightings would stay inside one continuous stretch instead of being counted as separate things. In other words, I was no longer looking at a pile of random snapshots. I was looking at movement. Once that rule was in place, the rest of the questions started making sense. I could count runs instead of rows, compare landings and takeoffs, look at how long an aircraft stayed in the airspace, and see when the traffic was building up or dropping off. That was the point where the data stopped feeling messy and started feeling usable. That is the first step to answer the question, how many flights are there in Mumbai over 2026? ### How many flights are there in Mumbai over 2026? Once I started grouping those repeated sightings into flight runs, the question became much clearer. Instead of asking how many rows exist, I could ask how many distinct aircraft movements exist from 1st of January, February and March of 2026. That is the number I actually care about. It tells me how much traffic was visible, when it showed up, and how those movements were spread across the day. There are approximately `7,237` inferred flight runs for the 3 full days of the 2026. For the 12 full days of 2024, there were approximately `51,441` inferred flight runs, and for 2025 there were approximately `43,324`. For 2026, it is just the first quarter (3 days compared to 12 days of those earlier years, so a good balance). Now we can move into interesting questions ### When is Mumbai most busy? Once I had the raw snapshots grouped into flight runs and filtered down to the airport-proximate ones, the first thing I wanted to see was simple: when does Mumbai actually get busy? For a "24 hour day" chart, I used: - one row per hour - count movements for each sampled day + hour - then average those hourly counts across sampled days That gives us a number representing on a typical sampled day, how many movements happen at 07:00, 08:00, etc.? ![Landing Takeoff Hourly](https://meetgor-cdn.pages.dev/flight-observatory-mumbai/landing_takeoff_hourly.png) The answer is pretty clear. In this sample, the busiest hour is 07:00 IST, with 1,086 movements. Right behind it are 20:00 (1,074), 17:00 (1,073), 18:00 (1,069), 21:00 (1,044), 11:00 (1,043), 19:00 (1,038), and 22:00 (1,032). The quietest hour is 03:00, with just 246 movements. So the traffic is not flat at all. It has a shape, and that shape is very visible once you count the flights as runs instead of raw rows. However if we take the average across all the sampled days, we get the same picture. For construcing the query for getting this, we want to know when the airport is most active during the day, so we can: - Keeps only airport-proximate events (lat and lon) - Labels each event as landing or takeoff (based on speed and altitude) - Converts the time to IST - Groups by hour of day - Counts movements in each hour bucket ![Movement Hourly Average](https://meetgor-cdn.pages.dev/flight-observatory-mumbai/hourly_avg.png) If we take the median instead, then the busiest hours are 17:00 IST, with 30.46 movements, 18:00 IST with 30.54 movements and 07:00 IST with 30.17 movements that makes it very consistent to the average. The quietest hour is still 03:00 with just 7.69 movements. That gives us a good answer and the difference is honestly not that big. The busiest hour is 07:00 and 17:00 with summed average and median respectively, and quietest is 03:00. The summed view and the averaged view tell the same story, but from two slightly different angles. In the total sample, 07:00 is the busiest hour, followed closely by 20:00 and 17:00. In the average-per-day view, the evening shifts slightly ahead, with 17:00 and 18:00 taking the top two spots and 07:00 still staying near the top. The quietest hour is the same in both views: 03:00. So the exact peak changes a little, but the shape does not. Mumbai is still busiest in the morning and evening, and quietest in the late night. ### Which Airlines Show Up Most I want to know which airlines dominate the traffic. Once the rows were grouped into flight runs, I could stop looking at raw sightings and just count which airline codes kept appearing. I used the first part of the flight code as a rough airline label and counted the runs for each one. That makes the dominant carriers obvious instead of buried inside repeated observations. What I did was with a query: - Takes the grouped flight runs - Pulls the airline prefix from the flight code - Counts how many runs each prefix appears in What we get: - A dominance chart - It shows which carriers keep repeating in the sample - IndiGo and Air India should stand out immediately This was quite evident, as I have observed Mumbai's sky for a long. Indigo and Air India are firmly dominant across the board. However, there are some other interesting observations like Vistara Air, Akasa and Emirates. In earlier pre-covid era, it used to be Jet Airways, but now Indigo has taken the lead. The difference in the first 3 is quite a big one, and I think its right! Since Indigo and Air-India, not only in Mumbai, they are preferred over other domestic airlines. Some other notable ones down the order are `SIA` for Singapore Airlines, `DLH` as Lufthansa, and `BAW` as British Airways. They are not as heavily appearing as domestic ones, they are regulars on a routine, each day every day. ``` ┌──────────────┬───────────┐ │ airline_code │ movements │ │ varchar │ int64 │ ├──────────────┼───────────┤ │ IGO │ 55050 │ │ AIC │ 20892 │ │ VTI │ 9789 │ │ AKJ │ 6923 │ │ UAE │ 5057 │ │ SEJ │ 4869 │ │ AXB │ 4514 │ │ QTR │ 3728 │ │ GOW │ 2191 │ │ ETD │ 2186 │ │ IAD │ 2151 │ │ @@@ │ 1708 │ │ SVA │ 1660 │ │ ABY │ 1542 │ │ ETH │ 1504 │ │ OMA │ 1479 │ │ SDG │ 1448 │ │ GFA │ 1214 │ │ SIA │ 979 │ │ KAC │ 825 │ │ DLH │ 771 │ │ THY │ 754 │ │ MAS │ 745 │ │ BAW │ 721 │ │ FDB │ 705 │ └──────────────┴───────────┘ ``` ![Airline Breakdown](https://meetgor-cdn.pages.dev/flight-observatory-mumbai/airline_routine.png) A good validation query! It seems the data was correctly analysed. ### Which Aircraft Type Dominates This was something I didn't have much idea about. I was not well versed with flight models and their names. So I let it shape what we can find. ``` ┌────────────────────────────────────────────────┬───────────┐ │ aircraft_type │ movements │ │ varchar │ int64 │ ├────────────────────────────────────────────────┼───────────┤ │ A20N (Airbus A320neo) │ 49269 │ │ A21N (Airbus A321neo) │ 27739 │ │ B38M (Boeing 737 MAX 8) │ 13138 │ │ A320 (Airbus A320) │ 12587 │ │ B77W (Boeing 777-300ER) │ 7543 │ │ B738 (Boeing 737-800) │ 5643 │ │ B789 (Boeing 787-9 Dreamliner) │ 3162 │ │ A321 (Airbus A321) │ 2761 │ │ B788 (Boeing 787-8 Dreamliner) │ 2482 │ │ B77L (Boeing 777-200LR / 777 freighter family) │ 2428 │ │ A359 (Airbus A350-900) │ 2385 │ │ A333 (Airbus A333) │ 2067 │ │ AT76 (A319) │ 1862 │ │ A319 (Airbus A319) │ 1736 │ │ A388 (Airbus A388) │ 1648 │ └────────────────────────────────────────────────┴───────────┘ ``` ![Aircraft type breakdown](https://meetgor-cdn.pages.dev/flight-observatory-mumbai/aircraft_types.png) ### Where Landings Start The question here is not where the aircrafts lands on the runway. It is where the aircrafts start coming down. That is a different thing. If I want to find the landing corridor, I should not look for the final stop point. I should look for the first clear movement in each run when the altitude starts dropping in a way that looks like descent. That gives me the start of the landing path, which is what matters for the map. Once I have those descent-start points, the next step is simple: group them by location and altitude. If the same area keeps showing up, that means aircrafts are repeatedly beginning their descent there. That is how the east-side corridor and the Ghatkopar-area low altitudes start making sense. The map is not showing a single landing dot. It is showing the place where arrival begins to feel like arrival. The patterns for a sharp dip is observed near the airport only, but overall, the gradual dip is seen on the Navi Mumbai side which is very true. The aircrafts gradually lowering their speeds and altitude descending into the runway strip. - 19.09,73.02 is east of the airport, out toward the eastern Mumbai / Thane creek / Navi Mumbai corridor. - 19.09,73.08 and 19.09,73.10 are farther east, so they sit deeper into the Navi Mumbai / Airoli / Ghansoli / Thane-side approach corridor. ``` ┌────────────┬────────────┬────────────────┬────────────────────┬───────────────┐ │ lat_bucket │ lon_bucket │ descent_points │ avg_alt_ft │ median_alt_ft │ │ double │ double │ int64 │ double │ double │ ├────────────┼────────────┼────────────────┼────────────────────┼───────────────┤ │ 19.09 │ 73.08 │ 142 │ 3405.6338028169016 │ 3400.0 │ │ 19.09 │ 73.1 │ 135 │ 3652.962962962963 │ 3600.0 │ │ 19.09 │ 72.86 │ 133 │ 36.2406015037594 │ 0.0 │ │ 19.09 │ 73.02 │ 123 │ 2638.008130081301 │ 2625.0 │ │ 18.58 │ 73.94 │ 122 │ 2115.1639344262294 │ 2175.0 │ │ 19.09 │ 73.01 │ 119 │ 2439.285714285714 │ 2425.0 │ │ 19.09 │ 73.11 │ 118 │ 3747.4576271186443 │ 3750.0 │ │ 19.09 │ 73.09 │ 116 │ 3565.0862068965516 │ 3525.0 │ │ 19.09 │ 72.94 │ 113 │ 1342.9203539823009 │ 1350.0 │ │ 19.09 │ 73.12 │ 113 │ 3951.7699115044247 │ 3925.0 │ │ 19.09 │ 73.23 │ 108 │ 5678.7037037037035 │ 5675.0 │ │ 19.09 │ 73.24 │ 102 │ 5867.156862745098 │ 5850.0 │ │ 19.09 │ 73.13 │ 101 │ 4157.425742574257 │ 4175.0 │ │ 19.09 │ 73.07 │ 99 │ 3325.5050505050503 │ 3325.0 │ │ 19.09 │ 73.18 │ 97 │ 4990.721649484536 │ 5025.0 │ │ 19.09 │ 73.2 │ 97 │ 5232.216494845361 │ 5175.0 │ │ 19.09 │ 73.14 │ 96 │ 4284.635416666667 │ 4275.0 │ │ 19.09 │ 72.91 │ 94 │ 819.4148936170212 │ 862.5 │ │ 18.58 │ 74.01 │ 92 │ 3408.4239130434785 │ 3400.0 │ │ 19.09 │ 73.0 │ 92 │ 2288.586956521739 │ 2300.0 │ ... ... ``` ![Landing Density](https://meetgor-cdn.pages.dev/flight-observatory-mumbai/descent_density_east_bw.png) A clear strip can be seen with the aircrafts descending into the runway strip. Also the dip is more pronounced in the Navi Mumbai / Airoli / Ghansoli / Thane-side approach corridor. I also looked for a query on the Ghatkopar region, and it was amazing to see that it was so near. Around `19.08, 72.94` is Ghatkopar, and aircrafts have an approximate average of 800 feets. 90% of them are below 1,100 feet, that is crazy low. And that clearly explains why the strip straight to the East-West of the airport line (which some part of Ghatkopar falls into) doesn't have any tall towers. ``` ┌────────────────┬───────────────────┬───────────────┬────────────┬────────────────────┐ │ descent_points │ avg_alt_ft │ median_alt_ft │ p90_alt_ft │ p95_alt_ft │ │ int64 │ double │ double │ double │ double │ ├────────────────┼───────────────────┼───────────────┼────────────┼────────────────────┤ │ 395 │ 859.7544303797469 │ 775.0 │ 1175.0 │ 1332.4999999999989 │ └────────────────┴───────────────────┴───────────────┴────────────┴────────────────────┘ ``` Another good metric to verify the dataset is proving right. ### Where Takeoffs Start Takeoffs are the same idea in reverse. The question is not where the plane ends up in the sky. The question is where it starts climbing away from the airport. So instead of looking for a finished takeoff, I look for the first point in each run where altitude clearly begins to rise. That gives me the start of the departure path. Then I group those start points by location. If the same area keeps lighting up, that tells me where departures are really getting going. That is why the takeoff map is tighter and closer to the airport than the landing map. It is not trying to show the whole flight. It is only trying to show the first step out of the airport, which is the part that makes the pattern visible. The sharp climb starts very close to the airport itself, and that is what the takeoff pattern shows clearly. The aircrafts begin to gain altitude almost immediately after leaving the runway, so the takeoff strip stays tight around the airport side instead of spreading out far into the city. - `19.09,72.81` and `19.09,72.82` sit just west of the airport and catch the early climb corridor right after liftoff. This is the sea side, beyond the Juhu aerodrome, perfect. `19.09,72.78` to `19.09,72.80` are a little farther out, but they still belong to the same western departure corridor, showing that the climb pattern stays compact near the field. - `19.09,72.84`, `19.09,72.85`, `19.09,72.86`, and 19.09,72.87 are even closer to the airport edge, so they sit in the immediate departure strip where the climb becomes visible first. - ``` ┌────────────────────┬───────────────────┬────────────────┬────────────────────┬───────────────┐ │ lat_bucket │ lon_bucket │ takeoff_points │ avg_alt_ft │ median_alt_ft │ │ double │ double │ int64 │ double │ double │ ├────────────────────┼───────────────────┼────────────────┼────────────────────┼───────────────┤ │ 19.09 │ 72.81 │ 1473 │ 2092.6171079429737 │ 2050.0 │ │ 19.09 │ 72.82000000000001 │ 1446 │ 1889.3672199170123 │ 1825.0 │ │ 19.09 │ 72.84 │ 1374 │ 1094.5778748180494 │ 1075.0 │ │ 19.09 │ 72.85000000000001 │ 1329 │ 764.8796087283672 │ 725.0 │ │ 19.09 │ 72.83 │ 1208 │ 1518.4395695364237 │ 1450.0 │ │ 19.09 │ 72.87 │ 1079 │ 465.8016682113068 │ 200.0 │ │ 19.09 │ 72.8 │ 994 │ 2233.6770623742455 │ 2150.0 │ │ 19.09 │ 72.86 │ 848 │ 698.1816037735849 │ 400.0 │ │ 19.09 │ 72.79 │ 538 │ 2579.460966542751 │ 2525.0 │ │ 19.09 │ 72.78 │ 486 │ 2790.792181069959 │ 2750.0 │ │ 19.09 │ 72.77 │ 418 │ 3224.5215311004786 │ 3000.0 │ │ 19.09 │ 72.76 │ 331 │ 3665.7099697885196 │ 3475.0 │ │ 19.09 │ 72.75 │ 234 │ 3924.5726495726494 │ 3662.5 │ │ 19.09 │ 72.88 │ 206 │ 976.2135922330098 │ 200.0 │ │ 19.09 │ 72.74 │ 137 │ 4216.605839416058 │ 3875.0 │ │ 19.09 │ 72.73 │ 129 │ 4780.232558139535 │ 4925.0 │ │ 19.09 │ 72.72 │ 68 │ 4781.985294117647 │ 4837.5 │ │ 19.18 │ 72.74 │ 62 │ 7492.741935483871 │ 7325.0 │ │ 19.19 │ 72.75 │ 62 │ 7558.870967741936 │ 7525.0 │ │ 19.080000000000002 │ 72.73 │ 54 │ 5360.185185185185 │ 5162.5 │ ... ... ``` ![Takeoff Density](https://meetgor-cdn.pages.dev/flight-observatory-mumbai/takeoff_density_west_bw.png) There is no major analysis here, since it has plain ocean in front of it, we can't really track the location thereafter. We can just use the direction that we'll check separately. ### Frequency of landings and takeoffs I wanted the spacing between landings on their own, and takeoffs on their own. So I grouped the airport events by type, sorted them in time order, and measured the gap between one landing and the next landing, and one takeoff and the next takeoff. The median is the number to trust most here, because a few long gaps pull the average up. The process was simple: - Classify events as landing or takeoff - Sort by time inside each sampled day - Compare each event to the next event of the same type - Measure the minutes between them ![Landing and Takeoff Frequency](https://meetgor-cdn.pages.dev/flight-observatory-mumbai/landing_takeoff_frequency.png) In this sample, landings are spaced about 2.58 minutes apart at the median, and takeoffs about 3.0 minutes apart. What can we observe from this? - Landings are spaced a bit more tightly than takeoffs - Takeoffs have a slightly longer pause between consecutive events - The median is the number to trust most here, because a few long gaps pull the average up This is the answer I have been chasing all along and it truly feels great to see it that close. As children we had observed the landing frequency as around 2 minute 30 seconds, and it couldn't be more accurate than this. But the great thing about the BOM/VABB Airport is that it still holds true even after a decade. It truly is the single runway busiest airport. ### Landing-to-Takeoff Gaps I want to measure the time between a flight landed and another took-off (not the same aircraft), any flight, like just after a flight landed, how much time it took for another to takeoff. I think it has to be in the range of 1-2 minutes. That's the status BOM/VABB Airport has to be the busiest single run way airport. Let's check and validate it with data. To answer that, I look at the airport events in time order and take each landing as a starting point. Then I find the next takeoff that follows it and measure the time gap between the two. If that gap is small, it means the airport is keeping the movement going very quickly. If it is large, it means there was a longer pause before the next departure. The interesting part is not the average alone, but the shape of the gaps across the sample. In this sample, that gap is very short most of the time. The median is 1.5 minutes, and the average is about 2.1 minutes, across 5,297 landing-to-takeoff intervals. Most of the bars are packed into the first few minutes, which tells me the airport is turning movements over very quickly. The long tail is there too, but it is small. So the real story is not "there are a few huge delays" it is that the typical landing is followed by a takeoff almost immediately. ![Landing-to-Takeoff Turnaround Gap](https://meetgor-cdn.pages.dev/flight-observatory-mumbai/turnaround_gap_refresh_clean_horizontal.png) And another question with intuition validated with data. If you think about it for a while, the median is 1.5 minutes, if you do the math (60/1.5 = 40 movements / hour). That alligns well with the hourly average. And it even validates the historical peak of Mumbai of 51 movements with 1,036 daily movements, that number is not far. It is that the airport's fame comes from how little space there is between operations. The chart makes that visible: most of the time, the next takeoff follows a landing almost immediately, and that is exactly the kind of traffic that built BOM's reputation. ### Weekday Pattern This is a bit interesting question, since I honestly had no clue about the answer. I thought every day would have the same pattern. I was wrong. To get this, it was straight forward. I took the airport events I had already classified as landings and takeoffs, grouped them by weekday in Mumbai time, and counted how many movements landed in each bucket. That gives a simple answer to a simple question: which day of the week gets the most airport activity? This question is asking about the breakdown of the flights based on days of the week. ``` ┌─────────────┬──────────────┬──────────┬──────────┬───────────┐ │ weekday_num │ weekday_name │ landings │ takeoffs │ movements │ │ int64 │ varchar │ int128 │ int128 │ int64 │ ├─────────────┼──────────────┼──────────┼──────────┼───────────┤ │ 0 │ Sunday │ 1444 │ 1239 │ 2683 │ │ 1 │ Monday │ 1383 │ 942 │ 2325 │ │ 2 │ Tuesday │ 1695 │ 1337 │ 3032 │ │ 3 │ Wednesday │ 1748 │ 1053 │ 2801 │ │ 4 │ Thursday │ 1805 │ 1236 │ 3041 │ │ 5 │ Friday │ 1859 │ 1323 │ 3182 │ │ 6 │ Saturday │ 2789 │ 2070 │ 4859 │ └─────────────┴──────────────┴──────────┴──────────┴───────────┘ ``` ![Weekday Takeoff-Landing Flights Breakdown](https://meetgor-cdn.pages.dev/flight-observatory-mumbai/weekday_breakdown.png) The result is pretty clear in the sample. Saturday stands out as the busiest day by a wide margin, with 4,859 movements, followed by Friday at 3,182 and Thursday at 3,041. Monday is the quietest at 2,325. > Landings are higher than takeoffs on every weekday, but the weekend spike is the real signal here. So the sample does not look evenly spread across the week, it has a strong Saturday peak and a weaker Monday trough. I then looked into the dataset and tried to find the count of each day in the sample, since i had around ~70 days of data (1 day * 12 months * 6 years) - Saturday 12 - Sunday 11 - Wednesday 11 - Friday 11 - Tuesday 10 - Thursday 10 - Monday 8 And that can explain why Saturday is the busiest day. The sample is not evenly distributed, but it is more or less evenly distributed across the calendar. ### Hotspots / Direction The question here is when the aircrafts starts landing or taking off, what direction are they actually pointing? That's quite obvious right? But I want to identify a deeper pattern here, from where are they coming from and where are they heading towards. That matters because it tells you whether Mumbai's traffic is spread randomly across the compass or funneled through one stable corridor. If the airport is using the same flow again and again, the headings should cluster into a small set of directions instead of being evenly scattered. To get this, we can: - take each inferred flight run - classify as a landing or takeoff - capture the heading at the relevant point in the run. For landings, it uses the heading at the end of the run. For takeoffs, it uses the heading at the start. ![Landing Direction Map](https://meetgor-cdn.pages.dev/flight-observatory-mumbai/landing_direction_map.png) ![Takeoff Direction Map](https://meetgor-cdn.pages.dev/flight-observatory-mumbai/takeoff_direction_map.png) Those headings are then grouped into direction sectors and counted, which is what produces the direction chart. ![Landing Heading Histogram](https://meetgor-cdn.pages.dev/flight-observatory-mumbai/landing_heading_hist_vertical.png) ![Takeoff Heading Histogram](https://meetgor-cdn.pages.dev/flight-observatory-mumbai/takeoff_heading_hist_vertical.png) The result is very strong and very repetitive. Most landings cluster around 270°, with a smaller 90° cluster, and takeoffs are also dominated by 270°. In the heading data, the top landing sector is 270° with 14,316 points, and the top takeoff sector is also 270° with 10,848 points. That tells me the airport is not behaving like a random cloud of movement. It is using the same directional axis again and again, which is exactly why the hotspot map looks like a corridor instead of a scatterplot. ### Time in Airspace The question here is are the aircrafts just passing through, or do they stay around Mumbai for a while? The idea is simple, if landings take longer than takeoffs, that usually means approach and descent are keeping the aircraft in the airspace longer. One thing I wanted to know was how long aircraft actually stay around Mumbai once they enter the airport flow. ![Time in Airspace](https://meetgor-cdn.pages.dev/flight-observatory-mumbai/special_time_in_airspace.png) So I looked at the duration of each inferred run and compared landings, takeoffs, and the full set together. The pattern is pretty clear: landings stay in the airspace longer than takeoffs. The median landing run is 34.5 minutes, while the median takeoff run is 25.1 minutes. Across all airport events, the median is 27.3 minutes. That tells me the traffic is not just passing through fast. A lot of it is lingering long enough for the airspace to feel full, especially on the landing side. The interesting part here is not just that landings last longer than takeoffs. It is what that tells me about the way Mumbai moves aircraft through its airspace. A landing is not a single movement. It is a long descent, even slower, and then the final stretch into the runway, so it naturally stays visible in the data for longer. Takeoffs are different. Once an aircraft leaves the runway and starts climbing out, it clears the airport zone much faster. That is why the landing median is 34.5 minutes, while takeoffs sit closer to 25.1 minutes. This does not mean the aircraft is simply flying slower in the sky. It means the arrival side of the airport keeps aircraft in the flow for longer, which is exactly what you would expect at a busy airport like Mumbai. It also hints at sequencing and approach management: arrivals are being held in the pattern of the airport longer than departures, which is the kind of behavior that makes the airspace feel crowded even before you count the total number of flights. So the real takeaway is this: Arrivals spend more time inside the airport's operating envelope, while departures clear out faster. That difference is part of the airport's style, and it helps explain why the airspace feels full even when the hourly flow looks ordinary on paper. ### Aircraft Turnaround Time > After this aircraft lands, how long does it stay on the ground before it takes off again? This is a good question, because it gives a deeper look at the activity on the airport. It doesn't necessarily give much but still a valid metric to observe. So I matched each landing with the next takeoff for the same aircraft on the same day. That gives a proper aircraft-level ground-time measure, instead of a flight-level handoff. ![Aircraft Turnaround Time](https://meetgor-cdn.pages.dev/flight-observatory-mumbai/special_same_aircraft_ground_gap.png) What I observed: The turnaround time is steady, but not extremely fast. The median ground time is 96.5 minutes, which is about 1 hour 37 minutes. The middle half of the aircraft stays between 77.5 and 130.5 minutes, and the 90th percentile goes up to 185.3 minutes. Most of the pairs sit in the 60-180 minute range, which suggests a working fleet that is being reused at a regular pace rather than left idle for long stretches. That also hints at the kind of airport Mumbai is. This looks much more like a high-frequency rotation airport than a place where aircraft disappear for half a day between uses. A lot of the traffic is probably made up of domestic and short-to-medium-haul operations, where the same aircraft can land, sit for a bit, and then go back out again within a couple of hours. So the takeaway is not just "there is a gap" It is that Mumbai keeps the aircraft moving in a fairly tight loop, which fits the airport's reputation for being heavily used and constantly in motion. ### International vs Domestic Flights I also wanted to see how much of Mumbai's traffic is Indian-registered versus foreign-registered. I could not see the route directly in the ADS-B feed, so I used aircraft registration as a proxy: `VT-` for Indian-registered aircraft and everything else as foreign-registered. That is not the same thing as domestic vs international routes, but it gives a useful high-level split. ``` ┌────────────────────────┬───────────┐ │ bucket │ movements │ │ varchar │ int64 │ ├────────────────────────┼───────────┤ │ Unknown │ 3399 │ │ Indian-registered (VT) │ 110492 │ │ Foreign-registered │ 37589 │ └────────────────────────┴───────────┘ ``` ![Domestic vs International Flight Aircraft Carrier Split](https://meetgor-cdn.pages.dev/flight-observatory-mumbai/special_domestic_international_proxy.png) The result is clear: about `72.9%` of movements are Indian-registered, while `24.8%` are foreign-registered. So the airport is still dominated by Indian traffic by count, but foreign aircraft are not a side note, or even a minority. They are a substantial and regular part of the flow. ## What Surprised Me What surprised me most was not that Mumbai is busy. I already expected that. What surprised me was how tight and consistent the whole system is. The airport does not just have a lot of movement, it has a deadly consistency to it. A landing is followed by a takeoff almost immediately, the same aircraft comes back into service in about an hour and a half, and the busiest hours are not random bursts but repeatable peaks that keep showing up in the same parts of the day. The other thing that stood out was how directional the traffic is. Once I started looking at landings and takeoffs on the map, the pattern stopped looking like scattered noise and started looking like a corridor. The airport is not using the full sky evenly. It keeps leaning on the same approach and departure axis, which makes the movement feel much more concentrated than a simple flight count would suggest. I mean that is no brainer, since it can disrupt the residential living, the noise and the unpredictable zoning of planes is not airports should be doing. And Mumbai does it very neatly. I also did not expect the weekday pattern to be so uneven. I assumed the traffic would be roughly similar across the week, but Saturday clearly pulled ahead in the sample. That was a good reminder that airports are not just shaped by the clock, they are also shaped by the calendar. Even with a dataset this limited, the weekly pattern still shows through. There is the big myth which gets debunked here, that is "Mumbai Airport is busiest in the late night". Opposite is true here. The airport is busiest in the morning and evening, and quietest in the late night. The airport is not empty for long. Though there might be international flights in the late night but they don't land and takeoff with the frequency as close as Indigo does, so that Mumbaikars can sleep finally. Another small but satisfying surprise was that the numbers matched the memory. As a child, I had the feeling that Mumbai's sky was never empty for long. The data backed that up. It felt satisfying to see the numbers in front of me for the frequency of landing, the takeoff to landing gaps, the dominance of airline, and mapping them to intuition and memories. The frequency is consistent like crazy, the turnaround period really tight, people do have a favorite preference of Indigo, everything just feels calm but its really quick and consistent. The airport really does stand up to its name "Single Runway Busiest Airport". ## Conclusion Well! That was a lot I admit. But it was fun. Atleast to me (I had said, it was a post for me!). I hope you got a little smile out of it if not just felt good reading about how Mumbai's Airport Operate. This is still ADS-B data, so it is a proxy, not an official operational log. But for the question I started with, it is enough. The feeling I had as a kid was real, Mumbai's sky was never empty for long. The frequency of landing is truly at 2 minute 30 second roughly speaking, which just makes me feel joyous, the airport turnaround time is quite a sharp and tight one, making it really a Mumbai-like busy runway. But that is just part 1 buddy! Haven't nerded about the complications of ADS-B data and my struggles of it, fighting with the limited disk space to compress the data and juggernauting queries to get results. That is the next part. Check out the full case study here -> [Flight Observatory - Mumbai Airspace Case Study](https://dev.meetgor.com/flight-observatory/case-study/mumbai-airport/)

Printf Debugging in Go with Q

Thu, 08 Jan 2026 00:00:00 UTC

This is cool, we make logging a mess. For logs we need to have separate scripts to get relevant data. How much chaos it can be.

Gotcha with Chained Assignment in Python

Wed, 27 Nov 2024 00:00:00 UTC

I was writing some Python code and wanted to initialize a few variables to an empty list. Instead of creating separate lists for each variable, I decided to use chained assignments like this: ```python a = b = c = [1,2,3] ``` It seems okay, nothing new, we are just assigning a, b, and c as an empty list. But notice the problem if you can. If we try to change the value of b, what do you think is going to happen? ```python b.append(4) ``` If you didn’t know about how Python handles assignments, you might expect the variables to behave like this: * `a` should still be `[1, 2, 3]` * `b` should be `[1, 2, 3, 4]` (since I modified b) * `c` should remain `[1, 2, 3]` ```python # You might think it will be this # a = [1,2,3] # b = [1,2,3,4] # c = [1,2,3] ``` But, hello python, it hides the pointer magic behind this statement The actual and expected state of the variables is: ```python # But it is actually this # a = [1,2,3,4] # b = [1,2,3,4] # c = [1,2,3,4] ``` When you chain the assignment like this, all three variables refer to the same list object in memory. This means, that all the variables will hold the same object, this means that if you change any of the variables assigned that reference it will change the object, and that will result in changing all the variables since they are referring to the same variable. You are not creating three independent lists. Instead, all three variables (a, b, and c) are referencing the same list object in memory. They don’t hold copies of the list, they all point to the same object. In Python, this is called **reference assignment**. When you modify one of the variables (like appending to b), you’re not modifying just b, you’re modifying the single list object that all three variables are referencing. Since all three variables point to the same list, any change you make to the list will be reflected in all three variables. On the other hand, if I do the same thing with strings, like this: ```python a = b = c = "hello" b = "world" # a = "hello" # b = "world" # c = "hello" ``` This only mutates the b variable and not the a and c, since string is not a mutable object in Python. In Python, the objects are either [mutable or immutable](https://realpython.com/python-mutable-vs-immutable-types/) Some of the primitive data types are: Mutable Types: * List * Dictionaries * Set * Byte Array Immutable Types: * Integer, Float, Complex * String * Tuple * Bytes * Boolean * Frozenset So, suppose you assign multiple variables with the same value of a mutable type. In that case, the change in one variable will mutate the other variables as well since the underlying object in memory is the same. So, this is what I learned from the mistake, avoid the chining assignment when dealing with mutable objects Instead do the following: ```python a, b, c = [], [], [] b.append(256) # a = [] # b = [256] # c = [] ``` This is safe and the right way to assign variables to individual values instead of the same value being referred to by all the variables. Happy Coding :)

Turn Python dictionary into a neat CSV table

Wed, 20 Mar 2024 00:00:00 UTC

## Populating a Python dict having a table-like structure to a CSV Today, I want to share with you a neat trick I recently discovered for populating a CSV file with data in a table-like structure using Python. ### Writing Key-Value Pairs to a CSV Row Firstly, let's discuss the `write_key_value` function. This function allows us to write key-value pairs to a CSV row. It's particularly useful when dealing with metrics or data that can be represented as simple pairs. ```python # Function to populate a CSV row with key-value pairs def write_key_value(writer, dictionary): for key, value in dictionary.items(): writer.writerow([key, value]) ``` ### Writing a Table-Like Structure to a CSV File Now, let's dive into the `write_table` function, which handles more complex scenarios where the data follows a table-like structure. This function takes into account different types of metrics and adjusts the CSV table structure accordingly. Assuming you have a structure of the dictionary like: ```python data = { "Students": { "John Doe": { "course": "Mathematics", "grade": "A", "attendance": 95, "assignments_completed": 15, "student_id": "JD001" }, "Alice Smith": { "course": "Physics", "grade": "B+", "attendance": 85, "assignments_completed": 12, "student_id": "AS002" }, "Bob Johnson": { "course": "Computer Science", "grade": "A-", "attendance": 90, "assignments_completed": 14, "student_id": "BJ003" } } } ``` And you want to write it to a CSV file, like this: ```csv student, course, grade, attendance, assignments_completed, student_id John Doe, Mathematics, A, 95, 15, JD001 Alice Smith, Physics, B+, 85, 12, AS002 Bob Johnson, Computer Science, A-, 90, 14, BJ003 ``` We can create a function `write_table` that will take in the `dictionary` as the actual data. We want to store the keys of the inner dictionary to be the header/columns of the csv file. As we can see the keys of the inner dict i.e. the value for the key `John Doe` is a dict with the keys `course`, `grade`, `attendance`, etc. which remain the same for the all the keys in the dictionary. So, we can first create a `row_keys` variable to store the keys of the actual dictionary this will be the first column rows in the csv. Further we check if the `row_keys` is a dict and then we append it with the `index_key` which will be the first column in the csv. Since all the keys remain the same for the inner-dict, we can pick the first dict and create the `header` with the inner-dict keys. So, we can write the list `header` to the csv file. Then for each key in the `row_keys` we can create a list `row` with the key and the values of the inner-dict. ```python # Function to populate a CSV with a table-like structure def write_table(writer, dictionary, index_key): row_keys = list(dictionary.keys()) if row_keys and data[row_keys[0]] is not None: headers = [index_key] + list( dictionary[row_keys[0]].keys() ) else: return writer.writerow(headers) for key in row_keys: row = [key] + list(dictionary[key].values()) writer.writerow(row) with open('data.csv', 'w', newline='') as csvfile: writer = csv.writer(csvfile) for key in data: write_table(writer, data[key], key) ``` ### Example Usage To illustrate how these functions can be used, let's consider a scenario where we have various types of metrics to populate into a CSV file. We handle key-value paired metrics separately and then populate the CSV with table-like metrics. ```python import csv data = { "Students": { "John Doe": { "course": "Mathematics", "grade": "A", "attendance": 95, "assignments_completed": 15, "student_id": "JD001" }, "Alice Smith": { "course": "Physics", "grade": "B+", "attendance": 85, "assignments_completed": 12, "student_id": "AS002" }, "Bob Johnson": { "course": "Computer Science", "grade": "A-", "attendance": 90, "assignments_completed": 14, "student_id": "BJ003" } }, "Countries": { "USA": { "capital": "Washington, D.C.", "population": 331000000, "area_sq_km": 9833517, "official_languages": ["English", "Spanish"], "currency": "United States Dollar (USD)" }, "India": { "capital": "New Delhi", "population": 1380004385, "area_sq_km": 3287263, "official_languages": ["Hindi", "English"], "currency": "Indian Rupee (INR)" }, "Brazil": { "capital": "Brasília", "population": 212559417, "area_sq_km": 8515770, "official_languages": ["Portuguese"], "currency": "Brazilian Real (BRL)" } } } def populate_table(writer, data, index_key): row_keys = list(data.keys()) if row_keys and data[row_keys[0]] is not None: headers = [index_key] + list(data[row_keys[0]].keys()) else: return writer.writerow(headers) for key in row_keys: row = [key] + list(data[key].values()) writer.writerow(row) with open('data.csv', 'w', newline='') as csvfile: writer = csv.writer(csvfile) for key in data: populate_table(writer, data[key], key) writer.writerow([]) ``` ```csv Students,course,grade,attendance,assignments_completed,student_id John Doe,Mathematics,A,95,15,JD001 Alice Smith,Physics,B+,85,12,AS002 Bob Johnson,Computer Science,A-,90,14,BJ003 Countries,capital,population,area_sq_km,official_languages,currency USA,"Washington, D.C.",331000000,9833517,"['English', 'Spanish']",United States Dollar (USD) India,New Delhi,1380004385,3287263,"['Hindi', 'English']",Indian Rupee (INR) Brazil,Brasília,212559417,8515770,['Portuguese'],Brazilian Real (BRL) ``` Thank you, Happy Coding :)

Create a Non-Clustered Index in Django with Postgres as DB

Thu, 10 Nov 2022 00:00:00 UTC

## What is a non-clustered index? A non-clustered index is a seperate structure than an actual table in the database, it stores the non-clustered index key(the column which we want to sort in the table), and a pointer to the actual values based on the index key. So, non-clustered indexes do not change the physical order of the table records, instead it holds a structure that can provide a easier and distinct way to fetch objects based on a particular column as the primary key in the structure. ## How to create a non-clustered index in django In django, we can use the [db_index](https://docs.djangoproject.com/en/4.1/ref/models/indexes/) property on a field(s) to create a index on the table/model. ### Add the property to the field in the model Chose a field in which, you want to add a index. It can be a foreign key or any other normal field defined in your model. We have used the typical blog model, so used in the some of my [TILS](https://www.meetgor.com/tils/) in django, it is just convenient to explain and understand as well. We have a django project named `core` and it has a app `blog` with a model defined below. The model `Article` has a few attributes like `title`, `description`, `content` and `status`. ```python from django.db import models ARTICLE_STATUS = [ ("PUBLISHED", "Published"), ("DRAFT", "Draft"), ] class Article(models.Model): title = models.CharField(max_length=128, db_index=True) description = models.CharField(max_length=512) content = models.TextField() status = models.CharField(max_length=16, choices=ARTICLE_STATUS, default="DRAFT") def __str__(self): return self.title ``` So, we have added a `db_index` to the title column in the model as a property. This will be equivalent to creating a index in `SQL` as follows: ``` $ python manage.py makemigrations Migrations for 'blog': blog/migrations/0002_alter_article_title.py - Alter field title on article ``` ``` $ python manage.py migrate Operations to perform: Apply all migrations: admin, auth, blog, contenttypes, sessions Running migrations: Applying blog.0002_alter_article_title... OK ``` Indexes are not standard as in SQL, but each vendor(sqlite, postgres, mysql) have their own flavour of syntax and naunces. ```sql CREATE INDEX "blog_article_title_3c514952" ON "blog_article" ("title"); CREATE INDEX "blog_article_title_3c514952_like" ON "blog_article" ("title" varchar_pattern_ops); ``` The above index commands are specific to the field, as the title field is a varchar, it has two types of index, it can generate one with simple match and other for `LIKE` comparisons because of string comparison behaviour. So, we just created a simple index and now if we query the db for a particular `title` which now has its own index for the table `blog_article`. This means, we will be able to fetch queries quickly if we are specifically filtering for `title`. ### Adding some data records We can add a few data records to test the query from the databse, you can ignore this part as it would be just setting up a django project and adding a few records to the databse. This part won't make sense for people reading to get the actual stuff done, move to the next part please. ``` python manage.py createsuperuser # Create a super user and run the server python manage.py runserver # Locate to http://127.0.0.1:8000/admin # Create some records in the artilce model ``` So, after creating some records, you should have a simple database and a working django application. ```sql SELECT * FROM blog_article; ``` ``` blog_test=# SELECT * FROM blog_article; id | title | description | content | status ----+----------+-------------+---------------------------+----------- 1 | test | test 1 | test content | DRAFT 2 | testpost | test 2 | test content more content | DRAFT 3 | newpost | test 3 | test nothing | PUBLISHED (3 rows) ``` ## Testing Queries We can now use SQL queries or django filters to check if we get results by a sequential or an index scan. If we have a filter of `title` we will get the results after performing an `Index Scan` which means, it will look up in the index columns rather than scanning the entire table of records. This is a way **we can test the indexes are working, efficiency is a differnet topic.** We can't get a idea of performance with this little data and just one connection. A real time database and having multiple conncurrent requests and connections is a good environment to test(don't do it in a production db :) ```sql EXPLAIN SELECT * FROM blog_article WHERE description LIKE 'test 2'; ``` ``` blog_test=# EXPLAIN ANALYSE SELECT * FROM blog_article WHERE description LIKE 'test'; --------------------------------------------------------------------------------------------------------- Seq Scan on blog_article (cost=0.00..11.00 rows=1 width=880) (actual time=0.180..0.181 rows=0 loops=1) Filter: ((description)::text ~~ 'test'::text) Rows Removed by Filter: 3 Planning Time: 0.189 ms Execution Time: 0.217 ms (5 rows) ``` The above query selects the records whose `description` is like `test 2`, this performs a `Sequenitial Scan` in the database i.e. iterating over the records one by one of the order of the primary key / id of the records in the table. ```sql EXPLAIN SELECT * FROM blog_article WHERE title LIKE 'test 2'; ``` ``` blog_test=# EXPLAIN ANALYSE SELECT * FROM blog_article WHERE title LIKE 'test'; --------------------------------------------------------------------------------------------------------- Index Scan using blog_article_title_3c514952_like on blog_article (cost=0.14..8.16 rows=1 width=880) (actual time=0.043..0.048 rows=1 loops=1) Index Cond: ((title)::text = 'test'::text) Filter: ((title)::text ~~ 'test'::text) Planning Time: 0.208 ms Execution Time: 0.093 ms (5 rows) ``` In the above query, the select statement has a filter with the title being like `test 2`, and since we have a index for looking for like of title column, the database performs a index scan on that table and fetches the result. Here are some tradeoffs, the planning is more and the execution time is less, this is quite logical as it would take time to make decision because the database has more options than before creating indexes. In the query where we filtered the description, the planning time was less as it makes sense there was just one option to go for sequential scan, but it took time to perform the operation as it would scan the entire table one by one. ## Using Django to test queries We can even use django to filter out the objects in the table. We simply use the `filter` method to check with a particular value. We can use the shell, to perform some queries. You can use this in your views or viewsets as per your requirements and constraints. We can even use `explain` to see what the underlying `sql` got executed out from the ORM. The [explain](https://docs.djangoproject.com/en/3.1/ref/models/querysets/#explain) function is similar to the `EXPLAIN ANALYSE` command in the `sql` queries. It gives a bit of context on how the query was executed. ``` $ python manage.py shell ``` ```python >>> from blog.models import Article >>> Article.objects.filter(description='test 1') ]> >>> Article.objects.filter(description='test 1').explain() "Seq Scan on blog_article (cost=0.00..11.00 rows=1 width=880)\n Filter: ((description)::text = 't est 1'::text)" >>> Article.objects.filter(title='test') ]> >>> Article.objects.filter(title='test').explain() "Index Scan using blog_article_title_3c514952_like on blog_article (cost=0.14..8.16 rows=1 width=8 80)\n Index Cond: ((title)::text = 'test'::text)" ``` We can use `__contains` for replicating the behaviour of `LIKE` in python/django from SQL. The below example will check if the title has a word `test` in any records of the database. ``` >>> Article.objects.filter(title__contains='test') , ]> ``` BONUS: We can even get the underlying SQL with the `.query.__str__()` method. ``` articles = Article.objects.filter(title__contains='test') articles.query.__str__() ``` ``` 'SELECT "blog_article"."id", "blog_article"."title", "blog_article"."description", "blog_article"." content", "blog_article"."status" FROM "blog_article" WHERE "blog_article"."title"::text LIKE %test %' ``` Here, we are able to see that clearly, that the django orm used the `LIKE` clause for comparing the title. Further readings and references: - [Indexing in Postgres](https://medium.com/geekculture/indexing-in-postgres-db-4cf502ce1b4e) - [Indexing refernece Django docs](https://docs.djangoproject.com/en/4.1/ref/models/indexes/) - [Non-Clustered indexing](https://gudevsoc.com/what-is-non-clustered-index-in-sql-with-example/)

Django Bulk Update QuerySet objects

Mon, 31 Oct 2022 00:00:00 UTC

Let's say, I have a lots of objects which I want to update with a particular field or fields. We can use the [bulk_update](https://docs.djangoproject.com/en/4.1/ref/models/querysets/#bulk-update) method with the model name. ```python # blog/models.py from django.db import models ARTICLE_STATUS = [ ("PUBLISHED", "Published"), ("DRAFT", "Draft"), ] class Article(models.Model): title = models.CharField(max_length=128) description = models.CharField(max_length=512) content = models.TextField() status = models.CharField(max_length=16, choices=ARTICLE_STATUS, default="DRAFT") def __str__(self): return self.title ``` Let's say we have a simple model `Article` with a few typical attributes like `title`, `description`, `content`, and `status`. We have the status as a choice field from two options as `Draft` and `Published`. It could be a boolean field, but that looks too gross for a article status. ```python from blog.models import Article articles = Article.objects.filter(status="draft") for i in range(len(articles)): articles[i].status = "published" Article.objects.bulk_update(articles, ["status",]) ``` In the above code, the `Articles` model is filtered by the status of `draft`. We iterate over the `QuerySet` which will contain the objects of the articles, by setting the object's properties to the value we want to set. We are jsut setting the value of the property of the object for each object. This just makes a changes to the `QuerySet`, by using the `bulk_update` method, the two parameters required are the `QuerySet` and the list of `fields` which are to be updated. The function returns the number of objects/records updated. ```python >>> from blog.models import Article >>> articles = Article.objects.filter(status="DRAFT") >>> articles , ]> >>> for i in range(len(articles)): ... articles[i].status = "PUBLISHED" ... >>> articles , ]> >>> >>> Article.objects.bulk_update(articles, ['status',]) 2 >>> Article.obejcts.get(title="test 1").status 'PUBLISHED' >>> Article.objects.filter(status="DRAFT") >>> ``` As, we can see here there were two obejcts `test 1` and `test 2` objects with the status as `Draft`. By iterating over the queryset and assigning the status of the object to published, the query set was changed and modified locally. By using the `bulk_update` method, we parsed the queryset and the list of attributes to be updated into the function. This gives us the number of objects which were updated, in this case `2`. We then look into the article actual record in the database and it has indeed updated to the value we set in this operation.

Python Pipreqs: Generate requirements file from the imported packages

Wed, 14 Sep 2022 00:00:00 UTC

## Introduction [Pipreqs](https://pypi.org/project/pipreqs/) is a python package that allows us to list all the pacakges which are imported in a python project. This is a great package for reducing the amount of redundant packages for a project. ## Install pipreqs You can install pipreqs with one of the many ways with pip, pipx, or any other pacakge management tool. I personally use pipreqs with `pipx` as it remains isolated from the rest of my project dependencies. ### Using simple pip install We can install with pip by creating a virtual environment or in a existing virtual environment. ``` pip install virtualenv venv source venv/bin/activate pip install pipreqs ``` ### Using pipx We can install pipreqs with pipx. [Pipx](https://pypi.org/project/pipx/) is also a python package but used as a tool to install any cli specific tool with the isolated environment. ``` pipx install pipreqs pipx run pipreqs ``` ## Using pipreqs We need to specify the encoding, which is used for reading the files while capturing the imports from the project. ``` pipx run pipreqs --encoding=utf-8 . ``` Additionaly, we can specify the `path` or filename where it will be used to save the imported packages. The `--savepath` option takes in the path to the file where you want to generate the list of the packages to be installed. ``` pipx run pipreqs --encoding=utf-8 --savepath reqs.txt . ``` Though this doesn't guarentee all the requirements for a file, it is really helpful for explicitly used packages in the python project.

Django-Mermaid

Wed, 03 Aug 2022 00:00:00 UTC

Create ER Diagrams (Entity Relationship Diagrams) of your Django projects related to different models of associated applications in Mermaid. Tech Stack: - Python - Django

Django: Get list of all models and associated fields in a django project

Tue, 02 Aug 2022 00:00:00 UTC

## Context Let's say we want the list of all the models and associated attributes in all the applications of a django project, we can do that using the [django.apps](https://docs.djangoproject.com/en/4.0/ref/applications/) with apps method. ## Get all the models in a project To fetch all the models, we can use the [get_models](https://docs.djangoproject.com/en/4.0/ref/applications/#django.apps.AppConfig.get_models) methods, it will return a list of model classes in all the entire project(all applications). We can import all the models in the django project with the command: ```python from django.apps import apps models = apps.get_models() ``` ``` [, , , , , , , , , , , , , , , ] ``` We are importing the apps and creating a list of the models in our django project. The Django app command will load all the applications in the project, and the [get_models]() method will fetch the associated models. This has resulted in a list of model class objects, we can iterate over them and fetch the required details, we want. For instance, If I am interested in the name of these models, I can use the `__name__` property to fetch the model's name. ```python from django.apps import apps model_list = apps.get_models() for model in model_list: print(model.__name__) ``` ``` LogEntry Permission Group ContentType Session EmailAddress EmailConfirmation SocialApp SocialAccount SocialToken TimeStampedModel User Tags Series Article Blog ``` So, from the above example, we can see we have accessed all the model names in our entire django project. ## Access Application name associated with a model For accessing the name of the application from the model class, we can use the `_meta` attribute followed by the `app_label` property to get the `app_name` associated with the model. ```python from django.apps import apps model_list = apps.get_models() for model in model_list: print(f"{model._meta.app_label} -> {model.__name__}") ``` ``` admin -> LogEntry auth -> Permission auth -> Group contenttypes -> ContentType sessions -> Session account -> EmailAddress account -> EmailConfirmation socialaccount -> SocialApp socialaccount -> SocialAccount socialaccount -> SocialToken user -> TimeStampedModel user -> User articles -> Tags articles -> Series articles -> Article blog -> Blog ``` In the above example, we can see we have printed all the models with their associated application names. ## Accessing all the attributes associated with a model To access all the fields/property/attributes associated with a model, we can again use the `_meta` attribute followed by the `get_fields` method. This method will return a list of field objects. For accessing the name of those attributes/fields, we have to iterate over the list and then further use `name` property. ```python from django.apps import apps model_list = apps.get_models() for model in model_list: print(model.__name__) field_list = model._meta.get_fields() for field in field_list: print(field.name) ``` ``` LogEntry id action_time user content_type object_id object_repr action_flag change_message Permission group user id ... ... Blog article id name description authors ``` So, that is how we get all the associated field names in the associated models in our django projects. Also, there are a lot of attributes, we can access with the apps property. The `__dict__.keys()` can be used to get the list of all associated properties or other methods in a class instance. ``` >>> m[14]._meta.get_fields()[4].__dict__.keys() dict_keys(['name', 'verbose_name', '_verbose_name', 'primary_key', 'max_length', '_unique', 'blank', 'null', 'remote_field', 'is_relation', 'default', 'editable', 'serialize', 'unique_for_date', 'unique_for_month', 'unique_for_year', 'choices', 'help_text', 'db_index', 'db_column', '_db_tablespace', 'auto_created', 'creation_counter', '_validators', '_error_messages', 'error_messages', 'db_collation', 'validators', 'attname', 'column', 'concrete', 'model']) ``` In the above example, I am using a list of models and getting the list of all the attributes associated with a field of a model. This can be applied and other properties can be accessed. ## Get Models with a specific app Let's say we want all the models associated with a particular application in the project, we can do that by specifying the name of the application. ```python from django.apps import apps app_info = apps.get_app_config('articles') print(app_info.__dict__.keys() print(app_info.verbose_name) print(app_info.models) print(app_info.models['article'].__dict__.keys()) ``` ``` dict_keys(['name', 'module', 'apps', 'label', 'verbose_name', 'path', 'models_module', 'models']) 'Article' {'tags': , 'series': , 'article': } dict_keys(['__module__', 'Article_Status', '__str__', 'get_absolute_url', '__doc__', '_meta', 'DoesNotExist', 'MultipleObjectsReturned', 'title', 'description', 'content', 'status', 'get_status_display', 'blog_id', 'blog', 'author_id', 'author', 'timestampedmodel_ptr_id', 'timestampedmodel_ptr']) ``` So, we can see that we have got the information about the app `articles` in the proejct where we can get the `verbose_name` property to fetch the human-readable format of the article model. Further, we can get all the models associated with the `articles` application. We get back a dict with the model name as the key and the class reference as the value. We have accessed the `article` model in the `articles` application and fetched all the associated properties or methods in the model. For further references, you can visit the [django apps documentation](https://docs.djangoproject.com/en/4.0/ref/applications/) to get more relevant methods and properties.

Django Blog DevLog: Load Frontmatter data into Template/Model Form Fields

Mon, 01 Aug 2022 00:00:00 UTC

## Introduction I have an Article Form where I load my post into it directly, it might have frontmatter. So what I wish to achieve is when I paste in the content, the frontmatter should be picked up and it should render the form fields like `title`, `description`, and then also remove the frontmatter from the content. To do that, we will require a model to work with and a form based on that model. We will exclude a few fields from that model so as to process these attributes on the server side. I am working on my Blog project which is a simple Django application. You can get the source code for the project on the [GitHub repository](https://github.com/mr-destructive/techstructive-blog/). ## Django Project Context The techstructive-blog is a django project, which has a couple of applications currently, not in a good situation. There are apps like `article`, `blog`, and `user`. This project has templates and static folder in the base directory. The project is deployed on [railway](https://www.railway.app) this is an always under development project, you can check out the [techstructive-blog](https://django-blog.up.railway.app). We can get the bits and pieces of the project details required for understanding what I want to do with the following sections. ### Article Model We have an `Article` model with attributes like `title`, `description`, `author` as a Foreign Key to the user model, and a few other attributes which is not related to what we are trying to achieve right now or at least don't require an explanation. We have a model method called `get_absolute_url` for getting a URL in order to redirect the client when the model instance is created or updated from the backend. You can definitely check out the details of these small components or templates in the project repository. ```python # articles/models.py class Article(TimeStampedModel): class Article_Status(models.TextChoices): DRAFT = ( "DRAFT", _("Draft"), ) PUBLISHED = ( "PUBLISHED", _("Published"), ) title = models.CharField(max_length=128) description = models.CharField(max_length=256) content = models.TextField(default="", null=False, blank=False) status = models.CharField( max_length=16, choices=Article_Status.choices, default=Article_Status.DRAFT, ) blog = models.ForeignKey(Blog, on_delete=models.CASCADE, null=True, blank=True) author = models.ForeignKey(User, on_delete=models.CASCADE, related_name="author") def __str__(self): return self.title def get_absolute_url(self): return reverse('articles:article-detail', args=[str(self.id)]) ``` In the below snippet, we have the forms defined in the article application for creating or updating of article instance. We will be using model forms as our form data should contain fields related to a model in this case the `Article` model. So when we inherit the `forms.ModelForm` in our custom `ArticleForm` we simply need to specify the model and it will take in all the attributes of that model by default, but if we specify the `fields` or `exclude` tuples, it will include only or exclude only the provided list of attributes from the model. We have also added the widgets for the form fields which will allow us to customize the way the fields in the template/form will render. We can specify the HTML attributes like `width`, `height`, `style`, etc. ### Article Form ```python # article/forms.py from django import forms from .models import Article class ArticleForm(forms.ModelForm): class Meta: model = Article exclude = ( "created", "updated", "author", ) widgets = { "title": forms.TextInput( attrs={ "class": "form-control", "style": "max-width: 450px; align: center;", "placeholder": "Title", } ), "description": forms.TextInput( attrs={ "class": "form-control", "style": "max-width: 900px; ", "placeholder": "Description", } ), "content": forms.Textarea( attrs={ "class": "form-control post-body", "id": "text-content", "style": "max-width:900px;", "hx-post": "/article/meta/", "placeholder": "Content", } ), "blog": forms.Select( attrs={ "class": "form-control", "placeholder": "Blog Publication", } ), } ``` So, these are my models and form files in the article app. Using htmx and without any javascript I want to update the form so that it picks up the front matter in the content field which is a text area and fills the title, description other attributes automatically for me. This can be done in a lot of ways, but I will be sharing one of the ways that I recently used in my blog project. This process involves creating a class-based view and adding a `POST` method that won't post any data to the backend but will send necessary data to the view. Let's see the process before diving into any of the code: ## Gist of the Process - Attach a `hx-post` attribute to the form field for sending the request to a view - When the request is sent, the data is loaded with `request.POST`, it is cleaned and converted in python-readable format with json. - Once we have the data, we try to use the `frontmatter.loads` function that will load the content and if we have a frontmatter in the text, it will load it as a `frontmatter.POST` object. - We will extract `title`, `description`, and other data fields from the object. - We will initialize a Form instance of Article, with the initial data values as the extracted data from the frontmatter. - Now we have two options: - If the article instance already exists i.e. we are updating the article - Else we are creating a new article Accordingly, we will load the form in the respective templates i.e. `update.html` for updating the existing articles and `article-form.html` for a new article. ## Adding HTMX Magic If you'd have seen we have a `hx-post` attribute in the `article/forms.py` file, the `content` widget has a `hx-post` attribute which sends a post request to the `article/meta/` URL route. This route we will bind to the `ArticleMetaView` which we will define in a few moments. This is usually sent once we change a certain text in the content field, so we can modify it as per your requirement with `hx-trigger` that can enable us to specify the trigger event or the type of trigger we want. Further, you can read from the [htmx docs](https://htmx.org/docs/#trigger-modifiers) about these triggers and other attributes. ```python # article/urls.py from django.urls import path from . import views app_name = "articles" urlpatterns = [ path("", views.ArticleCreateView.as_view(), name="article-create"), path("/", views.ArticleDetailView.as_view(), name="article-detail"), path("delete//", views.ArticleDetailView.as_view(), name="article-delete"), path("edit/", views.ArticleDetailView.as_view(), name="article-update"), # the new view that we will create path("meta/", views.ArticleMetaView.as_view(), name="article-meta"), ] ``` ## Capture Frontmatter Meta-data View Along with the Create, Detail/List, Update, Delete View, I will create a separate class called `ArticleMetaView` that will fetch the form fields and render the templates again but this time it will fill in the frontmatter meta-data in the fields if the content is parsed with the relvant frontmatter. ```python # articles/view.py class ArticleMetaView(View): model = Article def post(self, request, *args, **kwargs): data = json.loads(json.dumps(dict(request.POST))) loaded_frontmatter = frontmatter.loads(data['content'][0]) # frontmatter has keys i.e. attributes like title, description, etc. if dict(loaded_frontmatter): article_title = loaded_frontmatter['title'] article_description = loaded_frontmatter['description'] form = ArticleForm(initial={'title': article_title, 'description': article_description, 'content': loaded_frontmatter.content}) context = {'form': form} article_list = Article.objects.filter(title=article_title) if article_list: article = article_list.last() context['article'] = article return render(request, 'articles/edit_article.html', context) return render(request, 'articles/article_form.html', context) article_list = Article.objects.filter(title=data['title'][0]) # if the article title has been already taken i.e. we are updating an article if article_list: article = article_list.last() form = ArticleForm(data=request.POST) context = {'form': form} context['article'] = article return render(request, 'articles/edit_article.html', context) form = ArticleForm(data=request.POST) context = {'form': form} return render(request, 'articles/article_form.html', context) ``` In the above `ArticleMetaView` we have created a `post` method as we want to get hold of the content from the form. So, we start by extracting and converting the `request.body` data into an appropriate type for easily working with python. So, the `request.body` will contain the data like `csrf_token`, `form_data`, etc. received from the frontend template. We store the received data as `data` and now from this data, we can load the content field which will have the content information. Firstly we will extract the `request.body` which will contain the data from the form as we have made a `POST` request to this endpoint. For doing that we need to parse the content in a apropriate format such that it is python friendly. So we wrap the `request.body` into json format and then decode it back into the json string. This will give us the dict of the request data. ```python data = json.loads(json.dumps(dict(request.POST))) ``` ``` {'csrfmiddlewaretoken': ['bSYJxD39XH509tD1tZGd0WU21PUaKaLeqjjGbyzRvLXF4P8iIxb5l0fmTWVFjELQ'], 'title': ['test2'], 'description': ['test'], 'content': ['test something'], 'status': ['DRAFT'], 'blog': ['']} ``` So, this will grab the request data as a dict, we can then extract the data from this as it has data from the Form fields. We are interested in the content field in the Form, so we can get it by specifying the key `content` from the extracted data. But as we can see the data doesn't contain the actual data instead it is wrapped in a list i.e. `['test something']`, so we will have to index it and then fetch it. ```python content_string = data['content'][0] ``` This will give us the exact content field as a string. So, we can now move into extracting the frontmatter from the fields. Now, we can use the [frontmatter](https://python-frontmatter.readthedocs.io/en/latest/index.html) library to parse the content into the [loads](https://python-frontmatter.readthedocs.io/en/latest/api.html#frontmatter.loads) funciton and extract the frontmatter if it is present in the content field. The frontmatter library has a `loads` function which takes in a string and can give out a [frontmatter.Post](https://python-frontmatter.readthedocs.io/en/latest/api.html#post-objects) object. The loads function is differnet from the [load](https://python-frontmatter.readthedocs.io/en/latest/api.html#frontmatter.load) function as the load frunciton is for reading data from a stream of bytes i.e. a file or othe related byte object. The differnece is subtle but it took a read at the [documentation](https://python-frontmatter.readthedocs.io/en/latest/api.html#module-frontmatter). ```python post = data['content'][0] loaded_frontmatter = frontmatter.loads(post) ``` This wil load the content and give us a `frontmatter.Post` as said earlier. This will contain a dict with all the frontmatter if it has any and will by default parse the non-frontmatter data i.e. the remaining text into the `content` key. We need a chack if the Form field had any fronmatter this can be checked by the `dict(loaded_frontmatter)` which will return None if it cannot load the frontmatter. ```python loaded_frontmatter = frontmatter.loads(data['content'][0]) if dict(loaded_frontmatter): print(loaded_frontmatter.keys()) ``` ``` dict_keys(['templateKey', 'title', 'description', 'date', 'status']) ``` So once we have the frontmatter loaded we can get specific keys from it and initialize the form vaues to them. But we have made clear distictions that we want to perform a specific task if we have frontmatter keys in the content field of the Form else we can do something else. First let's handle the loading of the frontmatter into the form. For doing that we will get all the required attributes from the frontmatter like `title`, `description`, `content`, etc which can be accessed normally as we extract the value from a key in a dict. Once we have got those keys, we can start filling in the Form data with initial values. The [Django Model form](https://docs.djangoproject.com/en/4.0/topics/forms/modelforms/) takes in a parameter like [initial](https://docs.djangoproject.com/en/4.0/topics/forms/modelforms/#providing-initial-values) which can be a dict of the fiields along with the value that can be used for rendering the form initially when we load the template. ```python article_title = loaded_frontmatter['title'] article_description = loaded_frontmatter['description'] form = ArticleForm(initial={'title': article_title, 'description': article_description, 'content': loaded_frontmatter.content}) ``` This will take in a `ArticleForm` and fill the initial values like `title`, `description`, etc which we have provided in the dict with the values. Now, we need to parse this form in the current template or re-render the template. But before that, we need to parse this context into the template. We will create a dict with `form` as the key which can be used to render in the template. Also, we have a two ways here, either the user is creating a new article or it is updating a existing article. We need to make sure that we preserve the initial fields in the form as we are updating the existing article. So, we can filter the article objects as per the title of the current title and then if we find a article with that title, we will parse the context with that article object. ```python article_list = Article.objects.filter(title=article_title) if article_list: article = article_list.last() context{ 'form': form, 'article': article } return render(request, 'articles/edit_article.html', context) context = {'form': form} return render(request, 'articles/article_form.html', context) ``` Now, we have form data along with the article instance used for rendering the form with appropriate content. So, this will work for editing an already existing article. For a new article, we have to simply parse the form to the template and it will render the title picked from the fotnmatter or leave it empty. Similarly, for the article with no frontmatter we will iterate over the article and if the article's title already exist, we will render the article data with the form else render the form with the parsed title and other meta-data in the form. So that is how we render the form data with frontmatter into appropriate meta-data in the form. We have used Django forms and make use of HTMX for the dynamic updation of form.

APTUI

Wed, 22 Jun 2022 00:00:00 UTC

A terminal client interface for testing and working with APIs and web requests. Tech Stack: - Python - Textual - Rich

Autoformat Python file with Black after saving in Vim

Tue, 29 Mar 2022 00:00:00 UTC

If you are like me who writes Python very badly, it has empty lines with whitespaces, no proper format in assigning variables, not formatted according to [PEP 8](https://peps.python.org/pep-0008/) standards, and you use Vim as your text editor then my friend you need a autocmd badly for it. ## Install Black in Python Install the [black](https://pypi.org/project/black/) package in python globally or locally as per your preferences. ``` pip install black ``` OR with pipx ``` pipx install black ``` For a detailed guide about running packages with pipx head toward my article on [pipx](https://mr-destructive.github.io/techstructive-blog/pipx-intro/). ## Set up Autocmd in Vim We can set up a autocmd. What is a autocmd? It is about running commands when certain events occur like running a command when a file is saved, a buffer is opened or closed, and so on. What we want is to run the black command from the shell when the current file is saved. So, we can create a autocmd as follows: ```vimscript autocmd BufWritePost * !black % ``` Now, this will run when any type of file is saved, so we will make it specific to python by adding a `*.py` to add in the autocmd. ```vimscript autocmd BufWritePost *.py !black % ``` This works, but it gives a prompt after the command has been executed, to run the command silently we can simply add the silent keyword before the execution of the command from the shell. ```vimscript autocmd BufWritePost *.py silent !black % ``` This looks perfect! But still, we need to add a auto-group(`augroup`) that groups the autocmds and by adding `autocmd!` it will clear all the commands from the group. ```vimscript augroup python_format autocmd! autocmd BufWritePost *.py silent !black % augroup end ``` We can now add it to the vimrc to work all the time. ## Using pipx If you have used pipx to install black, you need to setup the autocmd a bit differently. ```vimscript autocmd BufWritePost *.py silent !pipx run black % ``` It might be a bit slower than running with global installation, but it is a neat way to run python package. So, that's it we can now write clean and safe python code without breaking a sweat in Vim. Happy Coding :)

Feedparser: Python package for reading RSS feeds

Sat, 26 Mar 2022 00:00:00 UTC

## Introduction [Feedparser](https://pypi.org/project/feedparser/) is a simple but powerful python package that can be used to extract information about a specific webpage or a publication with its RSS feed(not only RSS). By providing the RSS feed link, we can get structured information in the form of python lists and dictionaries. It can be basically used in a pythonic way to read RSS feeds, it is really simple to use and it even normalizes different types of feeds. Today, we will be taking a look at the feedparser package in python and how to extract information from a given RSS feed. ## What is feedparser Feedparser is a python package for parsing feeds of almost any type such as RSS, Atom, RDF, etc. It is a package that allows us to parse or extract information using python semantics. For example, all the latest posts from a given blog can be accessed on a list in python, further different attributes like links, images, titles, descriptions, can be accessed within a dictionary as key-value pairs. ## Installing feedparser As feedparser is a python package you can install it with pip very easily. ``` pip install feedparser ``` This will install feedparser in your respective python environment, it can be a virtual environment or a global environment. ## Using feedparser To test out feedparser, you can open up a python repl, in the environment where you installed the Feedparser package. ``` python ``` Firstly import the package. ```python import feedparser ``` Now, we can use the module in our application to get all of the functions or methods from the package. ## Parse an RSS feed URL To parse an RSS feed link, we can simply use the `parse` function from the feedparser package. The [parse](https://feedparser.readthedocs.io/en/latest/introduction.html) function takes in a string that can be a URL or a file path. Generally, the URL seems to be more useful. So, we can look up any RSS feed on the internet like your blog's feed, publications feeds, and so on. ```python feedparser.parse("url_of_the_rss_feed") ``` The parse function basically fetches the feed from the provided URL or the file. It extracts the feed in a systematic way storing each piece of information in a structured format. At the high level, it returns a dictionary with a few key-value pairs. Further, each key might have a list or nested dictionaries in it. The key identifiers are named in a uniform manner for any feed you parse in the function. Though there might be a few cases where there might be additional information to be parsed, it can even add more information ad shape the structure accordingly. This will give you a dictionary in python, that can have more or less similar keys. The most common keys that can be used in extracting information are `entries` and `feed`. We can get all the keys associated with a feed that is parsed using the `keys` function. ```python feedparser.parse("url_of_the_rss_feed").keys() ``` ![Feedparser Keys](https://res.cloudinary.com/techstructive-blog/image/upload/v1648370871/blog-media/ph6bsxobyifqmusumirx.png) The keys function basically gets all the keys in the dictionary in python. ``` >>> feedparser.parse("https://dev.to/feed/").keys() dict_keys(['bozo', 'entries', 'feed', 'headers', 'etag', 'href', 'status', 'encoding', 'version', 'namespaces']) ``` This will give out a list of all the keys in the feed which we have parsed from the RSS feed previously. From this list of keys, we can extract the required information from the feed. Before we extract content from the feed, we can store the dictionary that we get from calling the parse function. We can assign it to a variable and store the dictionary for later use. ```python feed = feedparser.parse("url_of_the_rss_feed") ``` ## Extract the contents from the feed Now, we have the dictionary of the feed, we can easily access the values from the listed keys. We can get the list of all the posts/podcasts/entries or any other form of content the feed is serving for from the `entries` key in the dictionary. To get more information and the most possible keys in the returned dictionary, you can refer to the feedparser [reference list](https://feedparser.readthedocs.io/en/latest/reference.html) ### Access Articles from Feed To access the articles from the feed, we can access those as a list of the articles. Using the `entries` key in the dictonary as follows: ```python feedparser.parse("url_of_the_rss_feed")["entries"] OR feedparser.parse("url_of_the_rss_feed").entries ``` If you have already defined a variable set to the parse function, you can use that for more efficient extraction. ```python feed = feedparser.parse("url_of_the_rss_feed") feed['entries'] OR feed.entries ``` ### Get Number of Articles/Entries from Feed To get the number of entries in the list, we can simply use the len function in python. ```python len(feed.entries) OR len(feedparser.parse("url_of_the_rss_feed").entries) ``` ![Feedparser Number of Entries](https://res.cloudinary.com/techstructive-blog/image/upload/v1648371042/blog-media/didijxcvsgvl4scrnhje.png) This gives us the number of entries in the provided feed. This is basically the list that stores all the content from the publication/website. So, we can iterate over the list and find all the different attributes we can extract. ### Get details of the entries from the feed To get detail information about a particular article/entry in the feed, we can iterate over the `feed.entries` list and access what we require. So, we will iterate over the entries and simply print those entries one by one to inspect what and how we can extract them. ```python for entry in feed.entries: print(entry) ``` It turns out that every entry in the list is a dictionary again containing a few key-value pairs like `title`, `summary`, `link`, etc. To get a clear idea of those keys we can again use the keys function in python. ```python feed = feedparser.parse("url_of_the_rss_feed") print(feed.entries[0].keys()) ``` ![Feedparser Entries Keys](https://res.cloudinary.com/techstructive-blog/image/upload/v1648371221/blog-media/c8uog85goe9jzrzl1pq1.png) ```python >>> feed.entries[0].keys() dict_keys(['title', 'title_detail', 'authors', 'author', 'author_detail', 'published', 'published_parsed', 'links', 'link', 'id', 'guidislink', 'summary', 'summary_detail', 'tags']) ``` Now, we have all the keys associated with the entries we can now extract the specific details like the content, like `title`, `author`, `summary_detail`(actual content in this case). Though this might not be the same for all RSS feeds, it might be very similar and a matter of using the right keyword for the associated keys in the list of dictionaries. Let's say, we want to print out the titles of all the entries in the feed, we can do that by iterating over the entries list and fetching the title from the iterator as `entry.title` if `entry` is the iterator. ```python for entry in feed.entries: print(entry.title) ``` ![Feedparser List of Entries](https://res.cloudinary.com/techstructive-blog/image/upload/v1648372532/blog-media/lhofdzmr3ks0fuut7pxm.png) Similarly, we will get the links of the entries using the link key in the dictionary. ```python for entry in feed.entries: print(entry.link) ``` ### Get information about the Website/Publication To get the metadata about the information you are extracting from i.e. the website or any publication, we can use the key `feed`. This key stores another dictionary as its value which might have information like `title`, `description` or `subtitle`, `canonical_url`, or any other data related to the website company. ```python feed.feed or feedparser.parse("url_of_the_rss_feed").feed ``` ![Feedparser Feed](https://res.cloudinary.com/techstructive-blog/image/upload/v1648373487/blog-media/r7hiojfdrtrjqfhkjbdt.png) From this dictionary, we can now simply extract the specific information from the keys. But first, as in the previous examples, we need a clear idea of what are the keys in the dictionary where we can extract the specific value. ```python feed.feed.keys() or feedparser.parse("url_of_the_rss_feed").feed.keys() ``` Using the keys like `title`, `links`, `subtitle`, we can get the information on the website/company level and not related to the specific post in the entries list. ```python # get the title of the webpage/publication feed.feed.title # get the links associated with the webpage feed.feed.links # get the cover-image for the webpage feed.feed.image ``` You can further get information specific to your feed. ## Checking for keys existence in the dictionary of feed We also need to check for the existence of a key in a dictionary in the provided feed, this can be a good problem if we are parsing multiple RSS feeds which might have a different structure. This problem occurred to me in the making of [podevcast](https://podevcast.netlify.app) where I had to parse multiple RSS feeds from different RSS generators. Some of the feeds had the cover image but most of them didn't. So, we need to make sure we have a check over those missing keys. ```python feedlist = ['https://freecodecamp.libsyn.com/rss', 'https://feeds.devpods.dev/devdiscuss_podcast.xml'] for feed in feedlist: feed = feedparser.parse(feed) print(feed.entries[0].keys()) for entry in feed.entries: if 'image' in entry: image_url = entry.image else: image_url = feed.feed.image #print(image_url) ``` ```python >>> feedlist = ['https://freecodecamp.libsyn.com/rss', 'https://feeds.devpods.dev/devdiscuss_podcast.xml'] >>> for feed in feedlist: ... feed = feedparser.parse(feed) ... for entry in feed.entries: ... if 'image' in entry: ... image_url = entry.image ... else: ... image_url = feed.feed.image ... print(feed.entries[0].keys()) ... dict_keys(['title', 'title_detail', 'itunes_title', 'published', 'published_parsed', 'id', 'guidislink', 'links', 'link', 'image', 'summary', 'summary_detail', 'content', 'itunes_duration', 'itunes_explicit', 'subtitle', 'subtitle_detail', 'itunes_episode', 'itunes_episodetype', 'authors', 'author', 'author_detail']) dict_keys(['title', 'title_detail', 'links', 'link', 'published', 'published_parsed', 'id', 'guidislink', 'tags', 'summary', 'summary_detail', 'content', 'subtitle', 'subtitle_detail', 'authors', 'author', 'author_detail', 'itunes_explicit', 'itunes_duration']) ``` As we can see we do not have an image key in the second RSS feed which means each entry doesn't have a unique cover image, so we have to fetch the image from the `feed` key then the `image` key in the entries list. ![Feedparser Cover Image Demo](https://res.cloudinary.com/techstructive-blog/image/upload/v1648373275/blog-media/fzdqie5dubigxzfhtv2x.png) As we can see here, the image_url will pick up the `image` key in the dictionary if it is present else we will assign it to another URL which is the website/podcast cover image. This is how we handle exceptions in providing the keys when there are multiple feeds to be extracted though they are quite similar, they will have subtle changes like this that need to be handled and taken care of. ## Conclusion From this little article, we were able to understand and use the feedparser Python package which can be used to extract information from different feeds. We saw how to extract contents for the entries, a number of entries in the feed, check for keys in the dictionary, and so on. Using Python's Feedparser package, some of the projects I have created include: - [podevcast](https://podevcast.netlify.app) - [dailydotdev-bookmark-cli](https://pypi.org/project/dailydotdev-bookmark-cli/) - [Django Newsletter](https://github.com/Mr-Destructive/newsletter) For further reading, you can specifically target a feed type by getting the appropriate methods from the feedparser [documentation](https://feedparser.readthedocs.io/en/latest/) Thank you for reading, if you have any suggestions, additions, feedback, please let me know in the comments or my social handles below. Hope you enjoyed reading. Happy Coding :)

Podevcast

Fri, 25 Feb 2022 00:00:00 UTC

Tech Stack: - Python - Python Pacakges - feedparser - Netlify - GitHub Actions

Crossposter

Fri, 25 Feb 2022 00:00:00 UTC

Tech Stack: - Python - Python Pacakges